Oct 29 21:43:39.250554 ip-10-0-128-85 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 29 21:43:39.250566 ip-10-0-128-85 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 29 21:43:39.250573 ip-10-0-128-85 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 29 21:43:39.250893 ip-10-0-128-85 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 29 21:43:49.499893 ip-10-0-128-85 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 29 21:43:49.499920 ip-10-0-128-85 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e46ec419cfbb4ee38e3df71bbcb4b970 --
Oct 29 21:45:43.400426 ip-10-0-128-85 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 21:45:43.766308 ip-10-0-128-85 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:45:43.766308 ip-10-0-128-85 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 21:45:43.766308 ip-10-0-128-85 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:45:43.766308 ip-10-0-128-85 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:45:43.769470 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769367    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:43.769470 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769470    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769475    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769479    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769482    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769485    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769488    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769490    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769493    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769496    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769499    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769501    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769504    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769506    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769509    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769512    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769514    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769517    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769519    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769522    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769525    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:43.769545 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769528    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769531    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769535    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769538    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769542    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769544    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769547    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769550    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769552    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769564    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769567    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769570    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769573    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769575    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769578    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769581    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769584    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769586    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769589    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:43.770096 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769591    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769593    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769596    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769598    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769601    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769604    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769606    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769609    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769612    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769615    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769617    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769620    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769622    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769625    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769628    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769631    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769635    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769637    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769640    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:43.770564 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769642    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769645    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769648    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769650    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769653    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769655    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769657    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769660    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769663    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769665    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769668    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769671    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769673    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769676    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769678    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769681    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769683    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769686    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769688    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769692    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:43.771013 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769694    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769697    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769700    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769702    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769705    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769707    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769710    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769713    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769716    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769718    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769721    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769723    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769726    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769728    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769731    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769734    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769736    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769740    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769743    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769745    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:43.771498 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769748    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769750    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769753    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769756    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769758    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769761    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.769763    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770591    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770599    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770602    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770605    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770608    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770611    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770614    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770618    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770621    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770623    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770626    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770629    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:43.771967 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770632    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770634    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770638    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770642    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770646    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770649    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770652    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770655    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770658    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770660    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770663    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770665    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770681    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770685    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770689    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770693    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770695    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770698    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770701    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:43.772484 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770703    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770706    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770709    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770711    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770714    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770716    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770719    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770721    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770724    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770727    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770730    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770732    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770735    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770738    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770741    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770743    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770746    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770748    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770751    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770753    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:43.772934 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770756    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770758    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770761    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770763    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770766    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770768    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770771    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770773    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770776    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770779    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770782    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770784    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770787    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770789    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770792    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770794    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770797    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770800    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770803    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:43.773426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770805    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770807    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770810    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770813    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770815    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770818    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770821    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770823    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770826    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770828    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770831    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770833    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770837    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770840    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770844    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770847    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770849    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770852    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770854    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:43.773868 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770857    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770859    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770862    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770864    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770867    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770871    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770874    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770876    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770879    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770881    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770883    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770889    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770891    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770894    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770896    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770899    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.770901    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771384    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771391    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771394    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:43.774334 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771396    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771399    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771402    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771405    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771408    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771410    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771413    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771416    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771418    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771421    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771423    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771426    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771429    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771431    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771434    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771436    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771439    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771441    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771444    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771446    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:43.774808 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771449    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771451    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771454    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771457    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771459    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771462    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771465    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771468    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771470    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771473    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771476    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771479    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771481    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771484    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771486    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771489    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771491    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771494    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771496    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771499    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:43.775283 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771502    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771504    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771507    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771509    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771512    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771515    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771517    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771520    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771522    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771525    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771527    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771530    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771532    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771535    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771537    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771540    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771543    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771545    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771548    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:43.775760 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771553    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771557    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771562    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771565    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771568    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771571    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771574    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771576    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771579    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771581    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771584    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771586    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771589    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771591    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771594    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771596    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771599    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771601    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771604    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:43.776208 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771606    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771609    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771611    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771614    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771616    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771619    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771621    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771624    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771626    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771629    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771634    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771636    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771639    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771641    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771644    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771646    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771649    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771652    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771654    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771658    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:43.776664 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771660    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771663    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771665    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771668    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.771670    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771768    2581 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771776    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771782    2581 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771786    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771791    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771794    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771799    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771802    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771806    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771809    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771812    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771815    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771818    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771821    2581 flags.go:64] FLAG: --cgroup-root=""
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771824    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771827    2581 flags.go:64] FLAG: --client-ca-file=""
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771830    2581 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771835    2581 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 21:45:43.777209 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771840    2581 flags.go:64] FLAG: --cluster-domain=""
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771843    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771846    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771850    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771853    2581 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771857    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771860    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771864    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771869    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771872    2581 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771875    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771878    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771881    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771884    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771889    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771891    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771894    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771898    2581 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771901    2581 flags.go:64] FLAG: --enable-server="true"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771904    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771908    2581 flags.go:64] FLAG: --event-burst="100"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771911    2581 flags.go:64] FLAG: --event-qps="50"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771914    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771917    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 21:45:43.777762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771920    2581 flags.go:64] FLAG: --eviction-hard=""
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771925    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771928    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771931    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771934    2581 flags.go:64] FLAG: --eviction-soft=""
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771937    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771940    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771944    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771947    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771950    2581 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771953    2581 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771956    2581 flags.go:64] FLAG: --feature-gates=""
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771960    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771963    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771967    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771970    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771976    2581 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771979    2581 flags.go:64] FLAG: --help="false"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771982    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-128-85.ec2.internal"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771986    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771989    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771991    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771995    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 21:45:43.778382 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.771998    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772001    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772004    2581 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772007    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772010    2581 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772013    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772017    2581 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772019    2581 flags.go:64] FLAG: --kube-reserved=""
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772022    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772025    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772028    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772031    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772034    2581 flags.go:64] FLAG: --lock-file=""
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772037    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772040    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772043    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772049    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772052    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772055    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772058    2581 flags.go:64] FLAG: --logging-format="text"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772061    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772065    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772068    2581 flags.go:64] FLAG: --manifest-url=""
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772071    2581 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772075    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 21:45:43.779290 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772080    2581 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772084    2581 flags.go:64] FLAG: --max-pods="110"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772088    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772091    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772094    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772097    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772100    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772103    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772106    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772112    2581 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772115    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772119    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772122    2581 flags.go:64] FLAG: --pod-cidr=""
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772125    2581 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772128    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772131    2581 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772134    2581 flags.go:64] FLAG: --port="10250"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772137    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772153    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-003a887b235a32254"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772157    2581 flags.go:64] FLAG: --qos-reserved=""
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772160    2581 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772163    2581 flags.go:64] FLAG: --register-node="true"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772166    2581 flags.go:64] FLAG: --register-with-taints=""
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772172    2581 flags.go:64] FLAG: --registry-burst="10"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772175    2581 flags.go:64] FLAG: --registry-qps="5"
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772178    2581 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 21:45:43.780363 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772180    2581 flags.go:64] FLAG: --reserved-memory=""
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772184    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772187    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772191    2581 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772194    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772196    2581 flags.go:64] FLAG: --runonce="false"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772200    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772204    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772211    2581 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772214    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772217    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772220    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772223    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772227    2581 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772230    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772233    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772236    2581 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772239    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772242    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772245    2581 flags.go:64] FLAG: --system-cgroups=""
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772248    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772253    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772256    2581 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772259    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772266    2581 flags.go:64] FLAG: --tls-min-version=""
Oct 29 21:45:43.781318 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772269    2581 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772272    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772275    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772278    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772282    2581 flags.go:64] FLAG: --v="2"
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772286    2581 flags.go:64] FLAG: --version="false"
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772291    2581 flags.go:64] FLAG: --vmodule=""
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772295    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772298    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772436    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772440    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772443    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772446    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772450    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772453    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772456    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772462    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772464    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772467    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772470    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772472    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772475    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:43.781987 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772478    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772480    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772483    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772485    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772488    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772491    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772494    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772496    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772499    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772502    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772504    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772507    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772510    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772512    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772517    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772519    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772523    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772527    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772531    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:43.782627 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772534    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772537    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772540    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772543    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772547    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772549    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772552    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772555    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772559    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772561    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772564    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772566    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772569    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772572    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772574    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772577    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772579    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772582    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772584    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772587    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:43.783467 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772589    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772592    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772595    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772597    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772600    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772602    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772605    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772609    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772611    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772614    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772616    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772619    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772621    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772624    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772626    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772629    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772633    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772636    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772639    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772641    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:43.784363 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772644    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772647    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772649    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772652    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772654    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772656    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772659    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772661    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772664    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772666    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772669    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772671    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772674    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772676    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772679    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772681    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772684    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772686    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772689    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:43.784930 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772694    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772696    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772700    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772704    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772707    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772710    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772712    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772715    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772717    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772721    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772724    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772726    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772729    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772732    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772734    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.772760    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:45:43.785713 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772801    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772805    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772808    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772811    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772813    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772816    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772819    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772822    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772825    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772827    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772830    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772832    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772835    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772837    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772840    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772843    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772845    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772849    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772852    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:43.786479 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772854    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772857    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772860    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772862    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772865    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772868    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772871    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772874    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772876    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772879    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772881    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772884    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772886    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772889    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772892    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772895    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772897    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772900    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772902    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772905    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:43.786973 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772907    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772910    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772912    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772915    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772917    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772920    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772924    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772927    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772930    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772932    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772937    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772940    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772943    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772945    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772948    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772951    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772953    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772957    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772961    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:43.787789 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772964    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772966    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772969    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772971    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772974    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772976    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772979    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772982    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772985    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772987    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772990    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772992    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772995    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.772997    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773000    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773002    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773005    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773007    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773010    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773012    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:43.788435 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773015    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773017    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773020    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773023    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773026    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773028    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773031    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773033    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773036    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773039    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773043    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773047    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773050    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773053    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773055    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773058    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773060    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773063    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773065    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:43.788909 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773068    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773071    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773073    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773076    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773078    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773082    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773084    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773087    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.773090    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.781022    2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.781042    2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.782438    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.782455    2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782541    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782549    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782555    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782560    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782565    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782569    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:43.789433 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782573    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782578    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782582    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782587    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782591    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782595    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782600    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782605    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782609    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782614    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782618    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782622    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782627    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782631    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782635    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782639    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782643    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782647    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782651    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782656    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:43.789876 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782660    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782664    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782669    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782675    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782680    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782684    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782688    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782692    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782696    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782700    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782705    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782709    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782714    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782718    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782722    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782726    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782731    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782735    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782740    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782744    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:43.790426 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782748    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782753    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782757    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782764    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782770    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782776    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782780    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782784    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782789    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782794    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782798    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782803    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782807    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782812    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782818    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782823    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782829    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782833    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:43.790878 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782838    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782843    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782847    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782851    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782856    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782861    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782866    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782870    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782874    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782879    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782883    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782887    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782891    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782896    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782900    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782904    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782909    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782913    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782917    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782922    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:43.791319 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782926    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782930    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782934    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782938    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782943    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782947    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782951    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782957    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782961    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782965    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782969    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782974    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782978    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782983    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782986    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782991    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782995    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.782999    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783003    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783007    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:43.791787 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783011    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783016    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.783054    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783288    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783297    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783301    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783306    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783310    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783314    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783319    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783323    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783327    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783331    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783335    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783340    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783344    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:43.792278 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783348    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783353    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783357    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783361    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783366    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783370    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783374    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783378    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783383    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783388    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783392    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783396    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783400    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783404    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783408    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783412    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783416    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783420    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783425    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783429    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:43.792684 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783433    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783437    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783442    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783448    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783455    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783461    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783466    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783470    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783474    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783477    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783481    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783485    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783489    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783494    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783498    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783503    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783507    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783511    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:43.793158 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783515    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783520    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783524    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783528    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783532    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783536    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783541    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783545    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783549    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783554    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783558    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783562    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783568    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783573    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783578    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783583    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783587    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783592    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783596    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:43.793593 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783600    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783604    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783608    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783612    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783617    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783620    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783626    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783630    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783635    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783639    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783643    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783647    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783651    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783656    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783660    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783664    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783668    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783673    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783678    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783691    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:43.794033 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783696    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783701    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783706    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783710    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783714    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783718    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783722    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783726    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783730    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783734    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783739    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783743    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783747    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783752    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783756    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783760    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:43.794508 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.783800    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783850    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783857    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783861    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783866    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783870    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783875    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783879    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783885    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783892    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783897    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783902    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783906    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783911    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783916    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783920    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783924    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783928    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:43.794881 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783933    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783937    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783941    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783945    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783949    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783954    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783958    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783962    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783966    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783970    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783975    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783980    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783984    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783988    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783992    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.783996    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784001    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784007    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784013    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:43.795388 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784018    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784024    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784029    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784034    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784038    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784042    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784046    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784051    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784055    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784060    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784065    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784072    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784076    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784080    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784085    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784089    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784093    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784097    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784101    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784106    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:43.795874 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784110    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784114    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784118    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784122    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784126    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784131    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784154    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784159    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784164    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784168    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784172    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784176    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784181    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784185    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784189    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784193    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784197    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784201    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784206    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784210    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:43.796356 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784215    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784219    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784223    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784229    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784234    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784238    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784243    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784249    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784253    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784257    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784261    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784265    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784270    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784274    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784278    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784282    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784286    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784290    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784296    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:43.796807 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784301    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784305    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784309    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784313    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784317    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784321    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784326    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784330    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784334    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784338    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:43.784342    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.784416    2581 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.787026    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.790570    2581 server.go:1014] "Starting client certificate rotation"
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.790670    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.791215    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.792550    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 21:45:43.797251 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.797104    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 21:45:43.806926 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.806910    2581 log.go:25] "Validated CRI v1 runtime API"
Oct 29 21:45:43.809894 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.809877    2581 log.go:25] "Validated CRI v1 image API"
Oct 29 21:45:43.811376 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.811345    2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 21:45:43.814829 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.814811    2581 fs.go:135] Filesystem UUIDs: map[22cb7a11-0711-4f85-925b-b6d85fc76f81:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d84f7fcd-5264-4fc0-ba4f-1c59059be07a:/dev/nvme0n1p4]
Oct 29 21:45:43.814899 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.814829    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 29 21:45:43.821320 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.821216    2581 manager.go:217] Machine: {Timestamp:2026-10-29 21:45:43.819793847 +0000 UTC m=+0.313127778 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100729 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bdd9549a8b7fce7274c90b34d505d SystemUUID:ec2bdd95-49a8-b7fc-e727-4c90b34d505d BootID:e46ec419-cfbb-4ee3-8e3d-f71bbcb4b970 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cd:67:d4:4b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cd:67:d4:4b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:d6:45:23:38:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 21:45:43.821790 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.821781    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 21:45:43.821869 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.821858    2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 21:45:43.825337 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.825312    2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 21:45:43.825459 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.825339    2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-85.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 21:45:43.825508 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.825462    2581 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 21:45:43.825508 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.825470    2581 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 21:45:43.825558 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.825502    2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 21:45:43.825558 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.825529    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 21:45:43.826357 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.826346    2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 21:45:43.830017 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.830005    2581 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 21:45:43.830163 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.830153    2581 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 21:45:43.830920 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.830910    2581 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 21:45:43.830951 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.830925    2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 21:45:43.830951 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.830934    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 21:45:43.830951 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.830942    2581 kubelet.go:405] "Adding apiserver pod source"
Oct 29 21:45:43.830951 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.830950    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 21:45:43.831399 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.831383    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 21:45:43.833553 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.833532    2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 21:45:43.839408 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.839392    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 21:45:43.840097 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.840084    2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 21:45:43.840162 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.840114    2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 21:45:43.840287 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.840276    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 21:45:43.840333 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.840292    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 21:45:43.840333 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.840298    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 21:45:43.840333 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.840304    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 21:45:43.840333 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.840310    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 21:45:43.840333 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.840316    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 21:45:43.840787 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.840777    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 21:45:43.840817 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.840787    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 21:45:43.840817 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.840794    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 21:45:43.840817 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.840802    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 21:45:43.840817 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.840810    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 21:45:43.840935 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.840820    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 21:45:43.842087 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.842078    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 21:45:43.842120 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.842088    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 21:45:43.846029 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.846017    2581 server.go:1263] "Started kubelet"
Oct 29 21:45:43.846130 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.846076    2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 21:45:43.846195 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.846122    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 21:45:43.846260 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.846196    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 21:45:43.846923 ip-10-0-128-85 systemd[1]: Started Kubernetes Kubelet.
Oct 29 21:45:43.848508 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.848023    2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 21:45:43.849530 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.849508    2581 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 21:45:43.851496 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:43.851475    2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 29 21:45:43.854008 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.853994    2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 21:45:43.854084 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.854003    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 21:45:43.854240 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:43.854222    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:45:43.854654 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.854538    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 21:45:43.854654 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.854548    2581 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 21:45:43.854654 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.854569    2581 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 21:45:43.854654 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.854639    2581 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 21:45:43.854654 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.854652    2581 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 21:45:43.855128 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.855107    2581 factory.go:55] Registering systemd factory
Oct 29 21:45:43.855128 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.855127    2581 factory.go:223] Registration of the systemd container factory successfully
Oct 29 21:45:43.855331 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.855320    2581 factory.go:153] Registering CRI-O factory
Oct 29 21:45:43.855372 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.855333    2581 factory.go:223] Registration of the crio container factory successfully
Oct 29 21:45:43.855401 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.855374    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 21:45:43.855401 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.855387    2581 factory.go:103] Registering Raw factory
Oct 29 21:45:43.855401 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.855397    2581 manager.go:1199] Started watching for new ooms in manager
Oct 29 21:45:43.856196 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.856185    2581 manager.go:319] Starting recovery of all containers
Oct 29 21:45:43.867548 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.867515    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 21:45:43.870212 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.870195    2581 manager.go:324] Recovery completed
Oct 29 21:45:43.874828 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.874816    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:43.876485 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.876469    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:43.876580 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.876502    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:43.876580 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.876518    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:43.877190 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.877174    2581 cpu_manager.go:226] "Starting" policy="none"
Oct 29 21:45:43.877190 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.877185    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:43.877190 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.877188    2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 21:45:43.877424 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.877209    2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 21:45:43.878569 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.878544    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-85.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 29 21:45:43.878671 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:43.878613    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-85.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 29 21:45:43.878919 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.878903    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:43.878993 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.878932    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:43.878993 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.878947    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:43.879380 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.879363    2581 policy_none.go:50] "Start"
Oct 29 21:45:43.879446 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.879385    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 21:45:43.879446 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.879400    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 21:45:43.879655 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:43.878547    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-85.ec2.internal.18e31e41a0f94cb1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-85.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-85.ec2.internal,},FirstTimestamp:2026-10-29 21:45:43.845989553 +0000 UTC m=+0.339323484,LastTimestamp:2026-10-29 21:45:43.845989553 +0000 UTC m=+0.339323484,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-85.ec2.internal,}"
Oct 29 21:45:43.881664 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.881650    2581 policy_none.go:44] "Start"
Oct 29 21:45:43.882988 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.882868    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 21:45:43.887344 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.887327    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cn5kg"
Oct 29 21:45:43.900323 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.900304    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cn5kg"
Oct 29 21:45:43.901596 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:43.901508    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-85.ec2.internal.18e31e41a2ca9fbe  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-85.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-85.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-85.ec2.internal,},FirstTimestamp:2026-10-29 21:45:43.876485054 +0000 UTC m=+0.369818992,LastTimestamp:2026-10-29 21:45:43.876485054 +0000 UTC m=+0.369818992,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-85.ec2.internal,}"
Oct 29 21:45:43.901694 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:43.901595    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 29 21:45:43.901694 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:43.901583    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-85.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 29 21:45:43.901694 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:43.901648    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 29 21:45:43.932909 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.911751    2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 21:45:43.932909 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.914231    2581 manager.go:341] "Starting Device Plugin manager"
Oct 29 21:45:43.932909 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:43.914253    2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 21:45:43.932909 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.914261    2581 server.go:87] "Starting device plugin registration server"
Oct 29 21:45:43.932909 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.914431    2581 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 21:45:43.932909 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.914441    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 21:45:43.932909 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.914562    2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 21:45:43.932909 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.914628    2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 21:45:43.932909 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.914635    2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 21:45:43.932909 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:43.915007    2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 21:45:43.932909 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:43.915054    2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:45:43.932909 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.921610    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 21:45:43.932909 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.921631    2581 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 21:45:43.932909 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.921650    2581 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 21:45:43.932909 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:43.921678    2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 29 21:45:43.936451 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:43.936425    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:44.015046 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.015025    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:44.016477 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.016428    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:44.016477 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.016471    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:44.016594 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.016484    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:44.016594 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.016507    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.022705 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.022689    2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal"]
Oct 29 21:45:44.022772 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.022754    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:44.024022 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.024008    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:44.024085 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.024036    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:44.024085 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.024059    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:44.026226 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.026214    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:44.026402 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.026389    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.026446 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.026428    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:44.026923 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.026904    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:44.026923 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.026920    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:44.027096 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.026934    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:44.027096 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.026938    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:44.027096 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.026948    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:44.027096 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.026948    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:44.029039 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.029023    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.029132 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.029047    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:44.029630 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.029617    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:44.029682 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.029644    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:44.029682 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.029655    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:44.035894 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.035877    2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.035987 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:44.035900    2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-85.ec2.internal\": node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:45:44.057570 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:44.057551    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-85.ec2.internal\" not found" node="ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.060627 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:44.060613    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-85.ec2.internal\" not found" node="ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.071528 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:44.071504    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:45:44.156694 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.156676    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc03cae4f4670aad3e1f4f4e61ad2796-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal\" (UID: \"cc03cae4f4670aad3e1f4f4e61ad2796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.156787 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.156701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc03cae4f4670aad3e1f4f4e61ad2796-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal\" (UID: \"cc03cae4f4670aad3e1f4f4e61ad2796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.156787 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.156716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d33928eb6a1a12294cb01f2c571c879b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-85.ec2.internal\" (UID: \"d33928eb6a1a12294cb01f2c571c879b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.171766 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:44.171748    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:45:44.257405 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.257384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc03cae4f4670aad3e1f4f4e61ad2796-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal\" (UID: \"cc03cae4f4670aad3e1f4f4e61ad2796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.257482 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.257428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc03cae4f4670aad3e1f4f4e61ad2796-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal\" (UID: \"cc03cae4f4670aad3e1f4f4e61ad2796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.257482 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.257445    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d33928eb6a1a12294cb01f2c571c879b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-85.ec2.internal\" (UID: \"d33928eb6a1a12294cb01f2c571c879b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.257482 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.257470    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d33928eb6a1a12294cb01f2c571c879b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-85.ec2.internal\" (UID: \"d33928eb6a1a12294cb01f2c571c879b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.257573 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.257475    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc03cae4f4670aad3e1f4f4e61ad2796-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal\" (UID: \"cc03cae4f4670aad3e1f4f4e61ad2796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.257573 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.257475    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc03cae4f4670aad3e1f4f4e61ad2796-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal\" (UID: \"cc03cae4f4670aad3e1f4f4e61ad2796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.272245 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:44.272202    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:45:44.360786 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.360765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.363686 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.363667    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal"
Oct 29 21:45:44.372946 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:44.372924    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:45:44.473234 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:44.473196    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:45:44.573789 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:44.573715    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:45:44.674226 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:44.674196    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:45:44.774781 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:44.774748    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:45:44.789917 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.789893    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 29 21:45:44.790043 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.790019    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 21:45:44.875280 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:44.875245    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:45:44.905094 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.905060    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 21:40:44 +0000 UTC" deadline="2028-07-25 18:51:44.071721044 +0000 UTC"
Oct 29 21:45:44.905094 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.905089    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15237h5m59.166635193s"
Oct 29 21:45:44.948489 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.948468    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:44.950722 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:44.950691    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc03cae4f4670aad3e1f4f4e61ad2796.slice/crio-c4b2ec9adbb917d9205f405a968e2677147436e0f71b66e5e17b1a27a7c548b4 WatchSource:0}: Error finding container c4b2ec9adbb917d9205f405a968e2677147436e0f71b66e5e17b1a27a7c548b4: Status 404 returned error can't find the container with id c4b2ec9adbb917d9205f405a968e2677147436e0f71b66e5e17b1a27a7c548b4
Oct 29 21:45:44.952898 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.952832    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:44.952898 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.952874    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:44.952898 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.952893    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:44.958555 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:44.958536    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 21:45:44.975368 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:44.975343    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:45:45.076458 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:45.076407    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:45:45.157241 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.157218    2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:45.176874 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:45.176848    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:45:45.274269 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.274250    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:45.329170 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.329086    2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:45.354421 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.354394    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal"
Oct 29 21:45:45.380637 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.380614    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:45:45.381743 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.381723    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:45:45.406683 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.406660    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:45:45.686303 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.686231    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:45.832274 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.832246    2581 apiserver.go:52] "Watching apiserver"
Oct 29 21:45:45.854605 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.854567    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 29 21:45:45.865030 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.865003    2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 21:45:45.865423 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.865399    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-t68qw","openshift-network-diagnostics/network-check-target-hvfh7","kube-system/global-pull-secret-syncer-7rvg5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal","openshift-network-operator/iptables-alerter-crv8b","kube-system/konnectivity-agent-4hxw6","openshift-cluster-node-tuning-operator/tuned-zqvj9","openshift-dns/node-resolver-6bf42","openshift-multus/multus-additional-cni-plugins-4kdn5","openshift-multus/multus-zmrms","openshift-ovn-kubernetes/ovnkube-node-mx2lj","kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs","openshift-image-registry/node-ca-rx95w"]
Oct 29 21:45:45.868829 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.868132    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.870979 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.870954    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:45.873020 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.872999    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:45.873122 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:45.873087    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rvg5" podUID="9c53ce77-00c9-4509-9650-115117f93d54"
Oct 29 21:45:45.876183 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.876139    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:45:45.878653 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.878628    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:45:45.881013 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.880994    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:45:45.883443 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.883423    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:45.886303 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.885570    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6chnm\""
Oct 29 21:45:45.886303 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.885705    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 21:45:45.886303 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.885719    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 21:45:45.886303 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.885737    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 21:45:45.886303 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.885766    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 21:45:45.886303 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.886277    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:45.887633 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.887514    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 21:45:45.890120 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.890093    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 21:45:45.890399 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.890381    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 21:45:45.890625 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.890607    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:45:45.890719 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:45.890700    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hvfh7" podUID="e5ac8782-ecd9-4940-85a8-af444cbb0a74"
Oct 29 21:45:45.890782 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.890712    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5fns5\""
Oct 29 21:45:45.893213 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.893192    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:45.893564 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:45.893383    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t68qw" podUID="aff21346-754f-4de1-b935-26f9a51145fe"
Oct 29 21:45:45.897372 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.897351    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.900166 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.900135    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:45:45.916317 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.916292    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 21:45:45.916417 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.916364    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8wwh7"
Oct 29 21:45:45.926912 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.926860    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal" event={"ID":"d33928eb6a1a12294cb01f2c571c879b","Type":"ContainerStarted","Data":"0d8115cff4151270a2a340c4c815e4edfa493fd575ee25208b986bad1f29f160"}
Oct 29 21:45:45.928453 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.928411    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal" event={"ID":"cc03cae4f4670aad3e1f4f4e61ad2796","Type":"ContainerStarted","Data":"c4b2ec9adbb917d9205f405a968e2677147436e0f71b66e5e17b1a27a7c548b4"}
Oct 29 21:45:45.930516 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.930494    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8wwh7"
Oct 29 21:45:45.956292 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.956274    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 21:45:45.966750 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.966728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-log-socket\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.966896 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.966875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-kubelet\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.967006 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.966908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:45.967006 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.966932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-registration-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:45.967134 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:45.967134 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b49ef0f6-2ac3-4393-b45a-5356110e13a3-konnectivity-ca\") pod \"konnectivity-agent-4hxw6\" (UID: \"b49ef0f6-2ac3-4393-b45a-5356110e13a3\") " pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:45:45.967134 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967109    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-var-lib-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.967134 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-sys-fs\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:45.967361 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/991476c7-c67c-436a-83e0-e1514350d019-tmp\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:45.967361 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysconfig\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:45.967361 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967308    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-node-log\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.967482 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967357    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.967482 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967420    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwqrl\" (UniqueName: \"kubernetes.io/projected/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-kube-api-access-kwqrl\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.967482 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scdz6\" (UniqueName: \"kubernetes.io/projected/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-kube-api-access-scdz6\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:45:45.967639 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:45.967639 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.967639 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-cnibin\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.967639 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-cni-bin\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.967833 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysctl-conf\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:45.967833 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09d024f0-dc43-4c99-81dd-ec9a2c126e86-hosts-file\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:45:45.967833 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d024f0-dc43-4c99-81dd-ec9a2c126e86-tmp-dir\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:45:45.967833 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsk5z\" (UniqueName: \"kubernetes.io/projected/09d024f0-dc43-4c99-81dd-ec9a2c126e86-kube-api-access-bsk5z\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:45:45.968019 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b49ef0f6-2ac3-4393-b45a-5356110e13a3-agent-certs\") pod \"konnectivity-agent-4hxw6\" (UID: \"b49ef0f6-2ac3-4393-b45a-5356110e13a3\") " pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:45:45.968019 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-run-netns\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.968019 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-etc-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.968019 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-ovn\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.968019 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.967975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-iptables-alerter-script\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:45:45.968019 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968011    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-lib-modules\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:45.968345 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-system-cni-dir\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:45.968345 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovnkube-script-lib\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.968345 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-device-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:45.968345 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppstg\" (UniqueName: \"kubernetes.io/projected/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-kube-api-access-ppstg\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:45:45.968345 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysctl-d\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:45.968345 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-host\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:45:45.968345 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968248    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-netns\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.968345 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-daemon-config\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.968345 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-host-slash\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:45:45.968803 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968378    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-modprobe-d\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:45.968803 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:45.968803 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8xqx\" (UniqueName: \"kubernetes.io/projected/aff21346-754f-4de1-b935-26f9a51145fe-kube-api-access-n8xqx\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:45.968803 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-cni-bin\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.968803 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-run\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:45.968803 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-var-lib-kubelet\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:45.968803 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cnibin\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:45.968803 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968624    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-cni-netd\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.968803 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968668    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovnkube-config\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.968803 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-host\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:45.968803 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovn-node-metrics-cert\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.968803 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-etc-kubernetes\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.968803 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-systemd\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/991476c7-c67c-436a-83e0-e1514350d019-etc-tuned\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968891    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-slash\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-systemd\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-kubelet\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.968978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjrdf\" (UniqueName: \"kubernetes.io/projected/e3d1ef12-c282-4cba-9678-e369daacc3d1-kube-api-access-mjrdf\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm6g9\" (UniqueName: \"kubernetes.io/projected/991476c7-c67c-436a-83e0-e1514350d019-kube-api-access-jm6g9\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-system-cni-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-cni-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-k8s-cni-cncf-io\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-conf-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4w7s\" (UniqueName: \"kubernetes.io/projected/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-kube-api-access-z4w7s\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-env-overrides\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-sys\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rlr5\" (UniqueName: \"kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5\") pod \"network-check-target-hvfh7\" (UID: \"e5ac8782-ecd9-4940-85a8-af444cbb0a74\") " pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.969487 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969334    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3d1ef12-c282-4cba-9678-e369daacc3d1-cni-binary-copy\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.970322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-socket-dir-parent\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.970322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-cni-multus\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.970322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c53ce77-00c9-4509-9650-115117f93d54-dbus\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:45.970322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969424    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:45.970322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-hostroot\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.970322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969496    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-multus-certs\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:45.970322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-socket-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:45.970322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:45.970322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-kubernetes\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:45.970322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-serviceca\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:45:45.970322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:45.970322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969619    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:45.970322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969642    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-systemd-units\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:45.970322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflhg\" (UniqueName: \"kubernetes.io/projected/12cd2b00-faff-422b-a588-c07050ab96f7-kube-api-access-rflhg\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:45.970322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.969736    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c53ce77-00c9-4509-9650-115117f93d54-kubelet-config\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:45.994909 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.994734    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:45:45.994909 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.994817    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 21:45:45.995197 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.994928    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 21:45:45.995197 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995007    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 21:45:45.995197 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995098    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 21:45:45.995367 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995270    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 21:45:45.995367 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995274    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:45:45.995367 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995298    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 21:45:45.995367 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995348    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l6fj9\""
Oct 29 21:45:45.995579 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995371    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 21:45:45.995579 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995436    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 21:45:45.995579 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995452    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 21:45:45.995579 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995527    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 21:45:45.995780 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995611    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8ld6j\""
Oct 29 21:45:45.995780 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995668    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 21:45:45.995780 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995669    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 21:45:45.995780 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995701    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7dqcg\""
Oct 29 21:45:45.995780 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995746    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 21:45:45.995780 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995744    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ckcdt\""
Oct 29 21:45:45.995780 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995749    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 21:45:45.996131 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995774    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 21:45:45.996131 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995822    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-znmmx\""
Oct 29 21:45:45.996131 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995883    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 21:45:45.996131 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.995962    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6jh7q\""
Oct 29 21:45:45.996131 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.996016    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 21:45:45.996467 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:45.996443    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mlr8g\""
Oct 29 21:45:46.001321 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.001296    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 21:45:46.070214 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm6g9\" (UniqueName: \"kubernetes.io/projected/991476c7-c67c-436a-83e0-e1514350d019-kube-api-access-jm6g9\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.070356 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-system-cni-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.070356 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-cni-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.070356 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070273    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-k8s-cni-cncf-io\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.070356 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-conf-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.070356 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4w7s\" (UniqueName: \"kubernetes.io/projected/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-kube-api-access-z4w7s\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:46.070356 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-env-overrides\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.070356 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-sys\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.070628 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlr5\" (UniqueName: \"kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5\") pod \"network-check-target-hvfh7\" (UID: \"e5ac8782-ecd9-4940-85a8-af444cbb0a74\") " pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:45:46.070628 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.070628 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3d1ef12-c282-4cba-9678-e369daacc3d1-cni-binary-copy\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.070628 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-socket-dir-parent\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.070628 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-cni-multus\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.070628 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c53ce77-00c9-4509-9650-115117f93d54-dbus\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:46.070628 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:46.070628 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070619    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-sys\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.070917 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-hostroot\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.070917 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-multus-certs\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.070917 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-socket-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:46.070917 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:46.070917 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-kubernetes\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.070917 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-serviceca\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:45:46.070917 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:46.070917 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:46.070917 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-systemd-units\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.070917 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflhg\" (UniqueName: \"kubernetes.io/projected/12cd2b00-faff-422b-a588-c07050ab96f7-kube-api-access-rflhg\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c53ce77-00c9-4509-9650-115117f93d54-kubelet-config\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070963    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-log-socket\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.070988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-kubelet\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-registration-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b49ef0f6-2ac3-4393-b45a-5356110e13a3-konnectivity-ca\") pod \"konnectivity-agent-4hxw6\" (UID: \"b49ef0f6-2ac3-4393-b45a-5356110e13a3\") " pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-var-lib-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071131    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-sys-fs\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/991476c7-c67c-436a-83e0-e1514350d019-tmp\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysconfig\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-node-log\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071312    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwqrl\" (UniqueName: \"kubernetes.io/projected/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-kube-api-access-kwqrl\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.071341 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scdz6\" (UniqueName: \"kubernetes.io/projected/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-kube-api-access-scdz6\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071403    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071429    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-cnibin\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-cni-bin\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysctl-conf\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071491    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-cnibin\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09d024f0-dc43-4c99-81dd-ec9a2c126e86-hosts-file\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d024f0-dc43-4c99-81dd-ec9a2c126e86-tmp-dir\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071553    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-system-cni-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsk5z\" (UniqueName: \"kubernetes.io/projected/09d024f0-dc43-4c99-81dd-ec9a2c126e86-kube-api-access-bsk5z\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b49ef0f6-2ac3-4393-b45a-5356110e13a3-agent-certs\") pod \"konnectivity-agent-4hxw6\" (UID: \"b49ef0f6-2ac3-4393-b45a-5356110e13a3\") " pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-run-netns\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-etc-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-ovn\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071698    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-cni-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.071886 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-iptables-alerter-script\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-lib-modules\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071749    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-k8s-cni-cncf-io\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-system-cni-dir\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071797    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-conf-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071803    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-system-cni-dir\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovnkube-script-lib\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-device-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppstg\" (UniqueName: \"kubernetes.io/projected/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-kube-api-access-ppstg\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysctl-d\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-host\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-netns\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.071982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-daemon-config\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-host-slash\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-modprobe-d\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8xqx\" (UniqueName: \"kubernetes.io/projected/aff21346-754f-4de1-b935-26f9a51145fe-kube-api-access-n8xqx\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:46.072562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-cni-bin\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-run\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-var-lib-kubelet\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cnibin\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-cni-netd\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovnkube-config\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-host\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovn-node-metrics-cert\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-etc-kubernetes\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-systemd\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/991476c7-c67c-436a-83e0-e1514350d019-etc-tuned\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-slash\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-systemd\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-kubelet\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrdf\" (UniqueName: \"kubernetes.io/projected/e3d1ef12-c282-4cba-9678-e369daacc3d1-kube-api-access-mjrdf\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072480    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-env-overrides\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072543    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-hostroot\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072595    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-multus-certs\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.073254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072694    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-socket-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:46.073918 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072813    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:46.073918 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.072867    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-kubernetes\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.073918 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.073140    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3d1ef12-c282-4cba-9678-e369daacc3d1-cni-binary-copy\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.073918 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.073208    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-socket-dir-parent\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.073918 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.073281    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-cni-multus\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.073918 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.073295    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-serviceca\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:45:46.073918 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.073374    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c53ce77-00c9-4509-9650-115117f93d54-dbus\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:46.073918 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.073411    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-cni-bin\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.073918 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.073452    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:46.073918 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.073513    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysctl-conf\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.073918 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.073545    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:46.073918 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.073555    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09d024f0-dc43-4c99-81dd-ec9a2c126e86-hosts-file\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:45:46.073918 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.073612    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs podName:aff21346-754f-4de1-b935-26f9a51145fe nodeName:}" failed. No retries permitted until 2026-10-29 21:45:46.573589848 +0000 UTC m=+3.066923774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs") pod "network-metrics-daemon-t68qw" (UID: "aff21346-754f-4de1-b935-26f9a51145fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:46.073918 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.073773    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d024f0-dc43-4c99-81dd-ec9a2c126e86-tmp-dir\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:45:46.073918 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.073793    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-systemd-units\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.074513 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.074068    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c53ce77-00c9-4509-9650-115117f93d54-kubelet-config\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:46.074513 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.074123    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-log-socket\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.074513 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.074169    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-kubelet\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.074513 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.074207    2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 21:45:46.074513 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.074244    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:46.074513 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.074300    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-registration-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:46.074513 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.074374    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:46.074513 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.074419    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret podName:9c53ce77-00c9-4509-9650-115117f93d54 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:46.57440433 +0000 UTC m=+3.067738255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret") pod "global-pull-secret-syncer-7rvg5" (UID: "9c53ce77-00c9-4509-9650-115117f93d54") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:46.074913 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.074622    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-host\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.074913 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.074856    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-cni-bin\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.074913 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.074906    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-run\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.074913 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.074892    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b49ef0f6-2ac3-4393-b45a-5356110e13a3-konnectivity-ca\") pod \"konnectivity-agent-4hxw6\" (UID: \"b49ef0f6-2ac3-4393-b45a-5356110e13a3\") " pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:45:46.075113 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.074937    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-var-lib-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.075113 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.074963    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-var-lib-kubelet\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.075113 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.075003    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-sys-fs\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:46.075113 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.075013    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cnibin\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:46.075113 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.075062    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-cni-netd\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.075509 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.075484    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysconfig\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.075610 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.075558    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-node-log\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.075670 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.075607    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.075775 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.075747    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:46.075855 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.075834    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-run-netns\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.075916 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.075890    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-etc-kubernetes\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.075998 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.075938    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-etc-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.076159 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.076032    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-ovn\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.076394 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.076358    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-systemd\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.076477 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.076427    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-slash\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.076535 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.076488    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-host\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:45:46.076660 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.076584    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-iptables-alerter-script\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:45:46.076660 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.076600    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-lib-modules\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.076660 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.076650    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-netns\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.077174 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.077149    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-daemon-config\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.077254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.077177    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovnkube-config\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.077254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.077198    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovnkube-script-lib\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.077254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.077235    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-host-slash\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:45:46.077254 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.075835    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.077401 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.077299    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-modprobe-d\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.077401 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.077306    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-systemd\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.077401 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.077302    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-device-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:46.077401 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.077351    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-kubelet\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.077569 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.077408    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysctl-d\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.077872 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.077848    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:46.078978 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.078950    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/991476c7-c67c-436a-83e0-e1514350d019-etc-tuned\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.079072 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.078958    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b49ef0f6-2ac3-4393-b45a-5356110e13a3-agent-certs\") pod \"konnectivity-agent-4hxw6\" (UID: \"b49ef0f6-2ac3-4393-b45a-5356110e13a3\") " pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:45:46.080075 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.080053    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/991476c7-c67c-436a-83e0-e1514350d019-tmp\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.080154 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.080123    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovn-node-metrics-cert\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.102510 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.102482    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:45:46.102510 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.102511    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:45:46.102671 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.102526    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2rlr5 for pod openshift-network-diagnostics/network-check-target-hvfh7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:46.102671 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.102602    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5 podName:e5ac8782-ecd9-4940-85a8-af444cbb0a74 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:46.602578569 +0000 UTC m=+3.095912506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2rlr5" (UniqueName: "kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5") pod "network-check-target-hvfh7" (UID: "e5ac8782-ecd9-4940-85a8-af444cbb0a74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:46.105410 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.105361    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4w7s\" (UniqueName: \"kubernetes.io/projected/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-kube-api-access-z4w7s\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:46.105483 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.105430    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsk5z\" (UniqueName: \"kubernetes.io/projected/09d024f0-dc43-4c99-81dd-ec9a2c126e86-kube-api-access-bsk5z\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:45:46.106296 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.106276    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflhg\" (UniqueName: \"kubernetes.io/projected/12cd2b00-faff-422b-a588-c07050ab96f7-kube-api-access-rflhg\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:46.112010 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.111990    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrdf\" (UniqueName: \"kubernetes.io/projected/e3d1ef12-c282-4cba-9678-e369daacc3d1-kube-api-access-mjrdf\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.112305 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.112279    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm6g9\" (UniqueName: \"kubernetes.io/projected/991476c7-c67c-436a-83e0-e1514350d019-kube-api-access-jm6g9\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.118436 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.118411    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scdz6\" (UniqueName: \"kubernetes.io/projected/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-kube-api-access-scdz6\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:45:46.119478 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.119431    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppstg\" (UniqueName: \"kubernetes.io/projected/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-kube-api-access-ppstg\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:45:46.122843 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.122809    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwqrl\" (UniqueName: \"kubernetes.io/projected/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-kube-api-access-kwqrl\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.123408 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.123389    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8xqx\" (UniqueName: \"kubernetes.io/projected/aff21346-754f-4de1-b935-26f9a51145fe-kube-api-access-n8xqx\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:46.181921 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.181892    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zmrms"
Oct 29 21:45:46.195696 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:46.195637    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d1ef12_c282_4cba_9678_e369daacc3d1.slice/crio-358fae070de9e341c80a1250438af79e63ec31b586d6cc2dff1073152278cba1 WatchSource:0}: Error finding container 358fae070de9e341c80a1250438af79e63ec31b586d6cc2dff1073152278cba1: Status 404 returned error can't find the container with id 358fae070de9e341c80a1250438af79e63ec31b586d6cc2dff1073152278cba1
Oct 29 21:45:46.198653 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.198627    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:45:46.204222 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.204166    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:45:46.211445 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.211426    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:45:46.212073 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:46.212029    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12cd2b00_faff_422b_a588_c07050ab96f7.slice/crio-bd77f956d4d408cdef442b8359f3a857b52b91945cbbd8648a6696c3a944d73b WatchSource:0}: Error finding container bd77f956d4d408cdef442b8359f3a857b52b91945cbbd8648a6696c3a944d73b: Status 404 returned error can't find the container with id bd77f956d4d408cdef442b8359f3a857b52b91945cbbd8648a6696c3a944d73b
Oct 29 21:45:46.216196 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:46.216171    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3cbc0d0_b4ec_4e1d_85c2_99f4642cd2f2.slice/crio-533f521b860f11ff4902df67f658c3bd6cdf11671160b32dee2732a566477840 WatchSource:0}: Error finding container 533f521b860f11ff4902df67f658c3bd6cdf11671160b32dee2732a566477840: Status 404 returned error can't find the container with id 533f521b860f11ff4902df67f658c3bd6cdf11671160b32dee2732a566477840
Oct 29 21:45:46.218948 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.218928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:45:46.221722 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:46.221696    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193ad4d9_4ef2_4ede_abd6_6ab5b589321a.slice/crio-cc38350aca267c87ffdd73d6f139252979c0d21a3017070c4ed216d04ba790db WatchSource:0}: Error finding container cc38350aca267c87ffdd73d6f139252979c0d21a3017070c4ed216d04ba790db: Status 404 returned error can't find the container with id cc38350aca267c87ffdd73d6f139252979c0d21a3017070c4ed216d04ba790db
Oct 29 21:45:46.230122 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:46.230100    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49ef0f6_2ac3_4393_b45a_5356110e13a3.slice/crio-eb90ff60287a8c76afe91c2493cd641894c8a83163328042c822928d2f886ff4 WatchSource:0}: Error finding container eb90ff60287a8c76afe91c2493cd641894c8a83163328042c822928d2f886ff4: Status 404 returned error can't find the container with id eb90ff60287a8c76afe91c2493cd641894c8a83163328042c822928d2f886ff4
Oct 29 21:45:46.242700 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.242676    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:45:46.244788 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.244765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:45:46.258770 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:46.258740    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9beffb_08c3_4766_bdd8_cdeaa557e5be.slice/crio-da2aa0ce7d11307030dcc28b4bcfdbf08454aabdc358d41f0aa2bdf913d8df92 WatchSource:0}: Error finding container da2aa0ce7d11307030dcc28b4bcfdbf08454aabdc358d41f0aa2bdf913d8df92: Status 404 returned error can't find the container with id da2aa0ce7d11307030dcc28b4bcfdbf08454aabdc358d41f0aa2bdf913d8df92
Oct 29 21:45:46.260098 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.259942    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:45:46.267595 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.267570    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:45:46.271888 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:46.271849    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d024f0_dc43_4c99_81dd_ec9a2c126e86.slice/crio-e6dd64c8ab1801f80a647e9603e93cf037daa5e56a3626056c0b2e9a8fdffc03 WatchSource:0}: Error finding container e6dd64c8ab1801f80a647e9603e93cf037daa5e56a3626056c0b2e9a8fdffc03: Status 404 returned error can't find the container with id e6dd64c8ab1801f80a647e9603e93cf037daa5e56a3626056c0b2e9a8fdffc03
Oct 29 21:45:46.279558 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:45:46.279531    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9b32d7_fb57_41c2_8b64_e5e4a2449169.slice/crio-cb7e36d875c48455585d287c440fe02e2ec49a2726b9c9de68581ba051e4445c WatchSource:0}: Error finding container cb7e36d875c48455585d287c440fe02e2ec49a2726b9c9de68581ba051e4445c: Status 404 returned error can't find the container with id cb7e36d875c48455585d287c440fe02e2ec49a2726b9c9de68581ba051e4445c
Oct 29 21:45:46.577273 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.577074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:46.577378 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.577297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:46.577378 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.577215    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:46.577464 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.577402    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:46.577464 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.577428    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs podName:aff21346-754f-4de1-b935-26f9a51145fe nodeName:}" failed. No retries permitted until 2026-10-29 21:45:47.577404559 +0000 UTC m=+4.070738482 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs") pod "network-metrics-daemon-t68qw" (UID: "aff21346-754f-4de1-b935-26f9a51145fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:46.577464 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.577457    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret podName:9c53ce77-00c9-4509-9650-115117f93d54 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:47.577442634 +0000 UTC m=+4.070776556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret") pod "global-pull-secret-syncer-7rvg5" (UID: "9c53ce77-00c9-4509-9650-115117f93d54") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:46.678317 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.678282    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlr5\" (UniqueName: \"kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5\") pod \"network-check-target-hvfh7\" (UID: \"e5ac8782-ecd9-4940-85a8-af444cbb0a74\") " pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:45:46.678463 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.678442    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:45:46.678463 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.678459    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:45:46.678591 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.678470    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2rlr5 for pod openshift-network-diagnostics/network-check-target-hvfh7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:46.678591 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.678527    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5 podName:e5ac8782-ecd9-4940-85a8-af444cbb0a74 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:47.67850816 +0000 UTC m=+4.171842098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rlr5" (UniqueName: "kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5") pod "network-check-target-hvfh7" (UID: "e5ac8782-ecd9-4940-85a8-af444cbb0a74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:46.922921 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.922795    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:46.923589 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:46.922942    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rvg5" podUID="9c53ce77-00c9-4509-9650-115117f93d54"
Oct 29 21:45:46.934840 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.934774    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:40:46 +0000 UTC" deadline="2028-06-23 03:56:10.480622689 +0000 UTC"
Oct 29 21:45:46.934840 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.934841    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14454h10m23.545787907s"
Oct 29 21:45:46.938754 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.938673    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zqvj9" event={"ID":"991476c7-c67c-436a-83e0-e1514350d019","Type":"ContainerStarted","Data":"a909920063c54a0129ef89e17b327bab051d6c70d2f204b76c6c63c4934f2e5a"}
Oct 29 21:45:46.948017 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.947988    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs" event={"ID":"12cd2b00-faff-422b-a588-c07050ab96f7","Type":"ContainerStarted","Data":"bd77f956d4d408cdef442b8359f3a857b52b91945cbbd8648a6696c3a944d73b"}
Oct 29 21:45:46.958456 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.958428    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zmrms" event={"ID":"e3d1ef12-c282-4cba-9678-e369daacc3d1","Type":"ContainerStarted","Data":"358fae070de9e341c80a1250438af79e63ec31b586d6cc2dff1073152278cba1"}
Oct 29 21:45:46.964014 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.963404    2581 generic.go:356] "Generic (PLEG): container finished" podID="cc03cae4f4670aad3e1f4f4e61ad2796" containerID="133ba2f2253133b948cf2f041a4adea29468a6eb5d958ae281d91cfc8338e78d" exitCode=0
Oct 29 21:45:46.964014 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.963655    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal" event={"ID":"cc03cae4f4670aad3e1f4f4e61ad2796","Type":"ContainerDied","Data":"133ba2f2253133b948cf2f041a4adea29468a6eb5d958ae281d91cfc8338e78d"}
Oct 29 21:45:46.970008 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.969946    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6bf42" event={"ID":"09d024f0-dc43-4c99-81dd-ec9a2c126e86","Type":"ContainerStarted","Data":"e6dd64c8ab1801f80a647e9603e93cf037daa5e56a3626056c0b2e9a8fdffc03"}
Oct 29 21:45:46.983337 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:46.983310    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kdn5" event={"ID":"bf9beffb-08c3-4766-bdd8-cdeaa557e5be","Type":"ContainerStarted","Data":"da2aa0ce7d11307030dcc28b4bcfdbf08454aabdc358d41f0aa2bdf913d8df92"}
Oct 29 21:45:47.002964 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:47.002892    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4hxw6" event={"ID":"b49ef0f6-2ac3-4393-b45a-5356110e13a3","Type":"ContainerStarted","Data":"eb90ff60287a8c76afe91c2493cd641894c8a83163328042c822928d2f886ff4"}
Oct 29 21:45:47.016072 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:47.016039    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-crv8b" event={"ID":"193ad4d9-4ef2-4ede-abd6-6ab5b589321a","Type":"ContainerStarted","Data":"cc38350aca267c87ffdd73d6f139252979c0d21a3017070c4ed216d04ba790db"}
Oct 29 21:45:47.020891 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:47.020862    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rx95w" event={"ID":"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2","Type":"ContainerStarted","Data":"533f521b860f11ff4902df67f658c3bd6cdf11671160b32dee2732a566477840"}
Oct 29 21:45:47.039073 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:47.039040    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal" event={"ID":"d33928eb6a1a12294cb01f2c571c879b","Type":"ContainerStarted","Data":"2460c3979229597df539441cc69fa3d1235d3b36646e237203a3b687e99827e3"}
Oct 29 21:45:47.043777 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:47.043749    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj" event={"ID":"7b9b32d7-fb57-41c2-8b64-e5e4a2449169","Type":"ContainerStarted","Data":"cb7e36d875c48455585d287c440fe02e2ec49a2726b9c9de68581ba051e4445c"}
Oct 29 21:45:47.585993 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:47.585898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:47.585993 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:47.585945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:47.586245 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:47.586075    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:47.586245 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:47.586132    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret podName:9c53ce77-00c9-4509-9650-115117f93d54 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:49.586114662 +0000 UTC m=+6.079448587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret") pod "global-pull-secret-syncer-7rvg5" (UID: "9c53ce77-00c9-4509-9650-115117f93d54") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:47.586383 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:47.586321    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:47.586383 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:47.586366    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs podName:aff21346-754f-4de1-b935-26f9a51145fe nodeName:}" failed. No retries permitted until 2026-10-29 21:45:49.586352104 +0000 UTC m=+6.079686030 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs") pod "network-metrics-daemon-t68qw" (UID: "aff21346-754f-4de1-b935-26f9a51145fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:47.687443 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:47.686808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlr5\" (UniqueName: \"kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5\") pod \"network-check-target-hvfh7\" (UID: \"e5ac8782-ecd9-4940-85a8-af444cbb0a74\") " pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:45:47.687443 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:47.687005    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:45:47.687443 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:47.687024    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:45:47.687443 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:47.687037    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2rlr5 for pod openshift-network-diagnostics/network-check-target-hvfh7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:47.687443 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:47.687096    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5 podName:e5ac8782-ecd9-4940-85a8-af444cbb0a74 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:49.687078311 +0000 UTC m=+6.180412236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rlr5" (UniqueName: "kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5") pod "network-check-target-hvfh7" (UID: "e5ac8782-ecd9-4940-85a8-af444cbb0a74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:47.926682 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:47.926601    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:47.927174 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:47.926756    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t68qw" podUID="aff21346-754f-4de1-b935-26f9a51145fe"
Oct 29 21:45:47.927174 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:47.927159    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:45:47.927333 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:47.927282    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hvfh7" podUID="e5ac8782-ecd9-4940-85a8-af444cbb0a74"
Oct 29 21:45:47.935034 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:47.934971    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:40:46 +0000 UTC" deadline="2028-07-26 08:43:47.836369328 +0000 UTC"
Oct 29 21:45:47.935034 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:47.935028    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15250h57m59.901345757s"
Oct 29 21:45:48.052194 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:48.052158    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal" event={"ID":"cc03cae4f4670aad3e1f4f4e61ad2796","Type":"ContainerStarted","Data":"627feaaa7af5d031b654f3cfb53f6e1be61041430e9006d7721de3833a647897"}
Oct 29 21:45:48.101163 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:48.101078    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal" podStartSLOduration=3.101058017 podStartE2EDuration="3.101058017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:47.086466733 +0000 UTC m=+3.579800680" watchObservedRunningTime="2026-10-29 21:45:48.101058017 +0000 UTC m=+4.594391962"
Oct 29 21:45:48.923343 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:48.922778    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:48.923343 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:48.922920    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rvg5" podUID="9c53ce77-00c9-4509-9650-115117f93d54"
Oct 29 21:45:49.607303 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:49.607218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:49.607303 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:49.607274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:49.607812 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:49.607431    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:49.607812 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:49.607498    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret podName:9c53ce77-00c9-4509-9650-115117f93d54 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:53.607478941 +0000 UTC m=+10.100812866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret") pod "global-pull-secret-syncer-7rvg5" (UID: "9c53ce77-00c9-4509-9650-115117f93d54") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:49.607931 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:49.607910    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:49.607979 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:49.607965    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs podName:aff21346-754f-4de1-b935-26f9a51145fe nodeName:}" failed. No retries permitted until 2026-10-29 21:45:53.607950226 +0000 UTC m=+10.101284149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs") pod "network-metrics-daemon-t68qw" (UID: "aff21346-754f-4de1-b935-26f9a51145fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:49.708357 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:49.707704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlr5\" (UniqueName: \"kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5\") pod \"network-check-target-hvfh7\" (UID: \"e5ac8782-ecd9-4940-85a8-af444cbb0a74\") " pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:45:49.708357 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:49.707902    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:45:49.708357 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:49.707921    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:45:49.708357 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:49.707934    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2rlr5 for pod openshift-network-diagnostics/network-check-target-hvfh7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:49.708357 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:49.707988    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5 podName:e5ac8782-ecd9-4940-85a8-af444cbb0a74 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:53.707971806 +0000 UTC m=+10.201305738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rlr5" (UniqueName: "kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5") pod "network-check-target-hvfh7" (UID: "e5ac8782-ecd9-4940-85a8-af444cbb0a74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:49.925189 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:49.925130    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:49.925288 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:49.925270    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t68qw" podUID="aff21346-754f-4de1-b935-26f9a51145fe"
Oct 29 21:45:49.925714 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:49.925597    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:45:49.925714 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:49.925693    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hvfh7" podUID="e5ac8782-ecd9-4940-85a8-af444cbb0a74"
Oct 29 21:45:50.073545 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:50.072711    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" containerID="cf6f712c999e6f5148e00c87356289885b68ab5f793704c9debd115cedf3fdce" exitCode=0
Oct 29 21:45:50.073545 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:50.072769    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kdn5" event={"ID":"bf9beffb-08c3-4766-bdd8-cdeaa557e5be","Type":"ContainerDied","Data":"cf6f712c999e6f5148e00c87356289885b68ab5f793704c9debd115cedf3fdce"}
Oct 29 21:45:50.139345 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:50.138383    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal" podStartSLOduration=5.138366142 podStartE2EDuration="5.138366142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:48.101790296 +0000 UTC m=+4.595124241" watchObservedRunningTime="2026-10-29 21:45:50.138366142 +0000 UTC m=+6.631700083"
Oct 29 21:45:50.921948 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:50.921916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:50.922400 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:50.922047    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rvg5" podUID="9c53ce77-00c9-4509-9650-115117f93d54"
Oct 29 21:45:51.078018 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:51.077980    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4hxw6" event={"ID":"b49ef0f6-2ac3-4393-b45a-5356110e13a3","Type":"ContainerStarted","Data":"a6367f37c1fb3d0e002a322e71b4115e4498038a5caac83d0410dc021ceb1e86"}
Oct 29 21:45:51.929614 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:51.929253    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:51.929614 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:51.929402    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t68qw" podUID="aff21346-754f-4de1-b935-26f9a51145fe"
Oct 29 21:45:51.929614 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:51.929482    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:45:51.929614 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:51.929554    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hvfh7" podUID="e5ac8782-ecd9-4940-85a8-af444cbb0a74"
Oct 29 21:45:52.083701 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:52.083613    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rx95w" event={"ID":"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2","Type":"ContainerStarted","Data":"607bedccf4e68ee4de1c0facae1edc89a6621a852b8d7855179bfeae417026d8"}
Oct 29 21:45:52.088329 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:52.088263    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs" event={"ID":"12cd2b00-faff-422b-a588-c07050ab96f7","Type":"ContainerStarted","Data":"bb07fa782e5bc3e27a968588cfb3032eb6e071b06854de44b1101772bba67561"}
Oct 29 21:45:52.127246 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:52.127113    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4hxw6" podStartSLOduration=4.458439356 podStartE2EDuration="8.127100394s" totalImagesPullingTime="3.668661038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:51.124712486 +0000 UTC m=+7.618046427" watchObservedRunningTime="2026-10-29 21:45:52.127100394 +0000 UTC m=+8.620434334"
Oct 29 21:45:52.922645 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:52.922567    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:52.922812 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:52.922716    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rvg5" podUID="9c53ce77-00c9-4509-9650-115117f93d54"
Oct 29 21:45:53.254993 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:53.254921    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:45:53.255648 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:53.255627    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:45:53.298548 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:53.298485    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rx95w" podStartSLOduration=4.389181516 podStartE2EDuration="9.29846539s" totalImagesPullingTime="4.909283874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:52.12774173 +0000 UTC m=+8.621075669" watchObservedRunningTime="2026-10-29 21:45:53.29846539 +0000 UTC m=+9.791799329"
Oct 29 21:45:53.649013 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:53.648391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:53.649013 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:53.648425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:53.649013 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:53.648545    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:53.649013 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:53.648649    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret podName:9c53ce77-00c9-4509-9650-115117f93d54 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:01.648615465 +0000 UTC m=+18.141949396 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret") pod "global-pull-secret-syncer-7rvg5" (UID: "9c53ce77-00c9-4509-9650-115117f93d54") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:53.649013 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:53.648943    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:53.649013 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:53.648987    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs podName:aff21346-754f-4de1-b935-26f9a51145fe nodeName:}" failed. No retries permitted until 2026-10-29 21:46:01.64897553 +0000 UTC m=+18.142309448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs") pod "network-metrics-daemon-t68qw" (UID: "aff21346-754f-4de1-b935-26f9a51145fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:53.691831 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:53.691583    2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 21:45:53.749778 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:53.749737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlr5\" (UniqueName: \"kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5\") pod \"network-check-target-hvfh7\" (UID: \"e5ac8782-ecd9-4940-85a8-af444cbb0a74\") " pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:45:53.749947 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:53.749931    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:45:53.750019 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:53.749949    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:45:53.750019 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:53.749961    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2rlr5 for pod openshift-network-diagnostics/network-check-target-hvfh7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:53.750019 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:53.750005    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5 podName:e5ac8782-ecd9-4940-85a8-af444cbb0a74 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:01.749993248 +0000 UTC m=+18.243327167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rlr5" (UniqueName: "kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5") pod "network-check-target-hvfh7" (UID: "e5ac8782-ecd9-4940-85a8-af444cbb0a74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:53.918439 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:53.918302    2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T21:45:53.691598194Z","UUID":"950dd0d8-f556-49e4-adb5-ab019fbee273","Handler":null,"Name":"","Endpoint":""}
Oct 29 21:45:53.925169 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:53.924946    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:45:53.925743 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:53.925481    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hvfh7" podUID="e5ac8782-ecd9-4940-85a8-af444cbb0a74"
Oct 29 21:45:53.926000 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:53.925373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:53.926000 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:53.925953    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t68qw" podUID="aff21346-754f-4de1-b935-26f9a51145fe"
Oct 29 21:45:53.930121 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:53.930098    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 21:45:53.930554 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:53.930536    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 21:45:54.097458 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:54.096788    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs" event={"ID":"12cd2b00-faff-422b-a588-c07050ab96f7","Type":"ContainerStarted","Data":"f64c97cd64c6852af103b9ef5176d43752e31abc171ba9466d10e97f7210f9e2"}
Oct 29 21:45:54.103613 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:54.103579    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zmrms" event={"ID":"e3d1ef12-c282-4cba-9678-e369daacc3d1","Type":"ContainerStarted","Data":"f24f9c31abadfd5c45514362a705205724e3b39e36b88680fe0ec099d2d891a7"}
Oct 29 21:45:54.149979 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:54.149916    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-zmrms" podStartSLOduration=2.347956931 podStartE2EDuration="10.149897344s" totalImagesPullingTime="7.801940413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:54.149112242 +0000 UTC m=+10.642446180" watchObservedRunningTime="2026-10-29 21:45:54.149897344 +0000 UTC m=+10.643231284"
Oct 29 21:45:54.921987 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:54.921958    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:54.922659 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:54.922084    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rvg5" podUID="9c53ce77-00c9-4509-9650-115117f93d54"
Oct 29 21:45:55.106511 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:55.106433    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6bf42" event={"ID":"09d024f0-dc43-4c99-81dd-ec9a2c126e86","Type":"ContainerStarted","Data":"0d07359c43390eeedac278be54b37edd653574a8b6cde7d3f2f79956c026efa6"}
Oct 29 21:45:55.109206 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:55.109173    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" containerID="0ab5015a1edf8b24fa2f2a0c931328d31abaff156dbc33d29f3bad611643dcfa" exitCode=0
Oct 29 21:45:55.109330 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:55.109257    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kdn5" event={"ID":"bf9beffb-08c3-4766-bdd8-cdeaa557e5be","Type":"ContainerDied","Data":"0ab5015a1edf8b24fa2f2a0c931328d31abaff156dbc33d29f3bad611643dcfa"}
Oct 29 21:45:55.111861 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:55.111838    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs" event={"ID":"12cd2b00-faff-422b-a588-c07050ab96f7","Type":"ContainerStarted","Data":"a76f68875dcfc0b4938766442397cd04fbc9f9976eaa4b067de7097db1b4b995"}
Oct 29 21:45:55.149517 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:55.149467    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6bf42" podStartSLOduration=3.4190681769999998 podStartE2EDuration="11.149449428s" totalImagesPullingTime="7.730381251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:55.149096762 +0000 UTC m=+11.642430716" watchObservedRunningTime="2026-10-29 21:45:55.149449428 +0000 UTC m=+11.642783371"
Oct 29 21:45:55.922450 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:55.922419    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:45:55.923060 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:55.922531    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hvfh7" podUID="e5ac8782-ecd9-4940-85a8-af444cbb0a74"
Oct 29 21:45:55.923060 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:55.922587    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:55.923060 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:55.922703    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t68qw" podUID="aff21346-754f-4de1-b935-26f9a51145fe"
Oct 29 21:45:56.115670 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:56.115618    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-crv8b" event={"ID":"193ad4d9-4ef2-4ede-abd6-6ab5b589321a","Type":"ContainerStarted","Data":"a7a4b827f9caa75bdbb2a124ecb2bdfd05a88863436ddfdd5aab34e8b763aaf4"}
Oct 29 21:45:56.160538 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:56.160485    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-crv8b" podStartSLOduration=4.293913896 podStartE2EDuration="12.160468431s" totalImagesPullingTime="7.866554535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:56.160347008 +0000 UTC m=+12.653680948" watchObservedRunningTime="2026-10-29 21:45:56.160468431 +0000 UTC m=+12.653802374"
Oct 29 21:45:56.161237 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:56.161202    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs" podStartSLOduration=4.076852369 podStartE2EDuration="12.161188751s" totalImagesPullingTime="8.084336382s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:55.245598731 +0000 UTC m=+11.738932674" watchObservedRunningTime="2026-10-29 21:45:56.161188751 +0000 UTC m=+12.654522693"
Oct 29 21:45:56.922331 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:56.922245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:56.922495 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:56.922374    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rvg5" podUID="9c53ce77-00c9-4509-9650-115117f93d54"
Oct 29 21:45:57.119513 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:57.119471    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zqvj9" event={"ID":"991476c7-c67c-436a-83e0-e1514350d019","Type":"ContainerStarted","Data":"772b5e07892938c31e94d4cf8338ef9d71b63f509408f21c180b8fa8e94aa8f3"}
Oct 29 21:45:57.122059 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:57.122031    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" containerID="750e8ac814df33179779b6d8bfc6417ff385a6d479d61696bb45ad6c69a548e3" exitCode=0
Oct 29 21:45:57.122200 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:57.122109    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kdn5" event={"ID":"bf9beffb-08c3-4766-bdd8-cdeaa557e5be","Type":"ContainerDied","Data":"750e8ac814df33179779b6d8bfc6417ff385a6d479d61696bb45ad6c69a548e3"}
Oct 29 21:45:57.165655 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:57.165604    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zqvj9" podStartSLOduration=3.019301213 podStartE2EDuration="13.165589686s" totalImagesPullingTime="10.146288473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:57.165373571 +0000 UTC m=+13.658707513" watchObservedRunningTime="2026-10-29 21:45:57.165589686 +0000 UTC m=+13.658923626"
Oct 29 21:45:57.922683 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:57.922487    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:45:57.923079 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:57.922550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:57.923079 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:57.922795    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hvfh7" podUID="e5ac8782-ecd9-4940-85a8-af444cbb0a74"
Oct 29 21:45:57.923079 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:57.922822    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t68qw" podUID="aff21346-754f-4de1-b935-26f9a51145fe"
Oct 29 21:45:58.127925 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:58.127834    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" containerID="c345f0962c58bcdc400cfd888c9f2ec90882afb7318f8928b312a352ad90bfaf" exitCode=0
Oct 29 21:45:58.128123 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:58.127919    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kdn5" event={"ID":"bf9beffb-08c3-4766-bdd8-cdeaa557e5be","Type":"ContainerDied","Data":"c345f0962c58bcdc400cfd888c9f2ec90882afb7318f8928b312a352ad90bfaf"}
Oct 29 21:45:58.922757 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:58.922726    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:45:58.923227 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:58.922856    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rvg5" podUID="9c53ce77-00c9-4509-9650-115117f93d54"
Oct 29 21:45:59.922390 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:59.921899    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:45:59.922390 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:45:59.921921    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:45:59.922390 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:59.922032    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t68qw" podUID="aff21346-754f-4de1-b935-26f9a51145fe"
Oct 29 21:45:59.922390 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:45:59.922110    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hvfh7" podUID="e5ac8782-ecd9-4940-85a8-af444cbb0a74"
Oct 29 21:46:00.134622 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:00.134589    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" containerID="e93981ed89981a2e3b96ed0c0e6e85fca8a4c900e53985ec48a4506985ba6ab9" exitCode=0
Oct 29 21:46:00.135036 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:00.134649    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kdn5" event={"ID":"bf9beffb-08c3-4766-bdd8-cdeaa557e5be","Type":"ContainerDied","Data":"e93981ed89981a2e3b96ed0c0e6e85fca8a4c900e53985ec48a4506985ba6ab9"}
Oct 29 21:46:00.922808 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:00.922781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:00.922965 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:00.922877    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rvg5" podUID="9c53ce77-00c9-4509-9650-115117f93d54"
Oct 29 21:46:01.139331 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:01.139178    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj" event={"ID":"7b9b32d7-fb57-41c2-8b64-e5e4a2449169","Type":"ContainerStarted","Data":"cb92a50c8b2165278af90886dcd378bf01656eac735387ce926087f25a2685b0"}
Oct 29 21:46:01.139807 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:01.139436    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj" event={"ID":"7b9b32d7-fb57-41c2-8b64-e5e4a2449169","Type":"ContainerStarted","Data":"d08d96b194d9577b993a4cda313a9f5d2e633ef1863aec7a0456d1b72bca2066"}
Oct 29 21:46:01.139807 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:01.139490    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj" event={"ID":"7b9b32d7-fb57-41c2-8b64-e5e4a2449169","Type":"ContainerStarted","Data":"4f31a81eaba91152cf97e54cffe9fe4ba14e40b1c9c964ee5c70ee4bf3d67bc5"}
Oct 29 21:46:01.142360 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:01.142333    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" containerID="3199d7abd32e1f5320633223311c9298783ace57d11588b2dcfc0d3eb12cab43" exitCode=0
Oct 29 21:46:01.142446 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:01.142376    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kdn5" event={"ID":"bf9beffb-08c3-4766-bdd8-cdeaa557e5be","Type":"ContainerDied","Data":"3199d7abd32e1f5320633223311c9298783ace57d11588b2dcfc0d3eb12cab43"}
Oct 29 21:46:01.709274 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:01.709244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:01.709457 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:01.709282    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:01.709457 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:01.709392    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:46:01.709457 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:01.709447    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret podName:9c53ce77-00c9-4509-9650-115117f93d54 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:17.709429698 +0000 UTC m=+34.202763634 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret") pod "global-pull-secret-syncer-7rvg5" (UID: "9c53ce77-00c9-4509-9650-115117f93d54") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:46:01.709620 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:01.709393    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:46:01.709620 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:01.709514    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs podName:aff21346-754f-4de1-b935-26f9a51145fe nodeName:}" failed. No retries permitted until 2026-10-29 21:46:17.709500609 +0000 UTC m=+34.202834527 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs") pod "network-metrics-daemon-t68qw" (UID: "aff21346-754f-4de1-b935-26f9a51145fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:46:01.809645 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:01.809620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlr5\" (UniqueName: \"kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5\") pod \"network-check-target-hvfh7\" (UID: \"e5ac8782-ecd9-4940-85a8-af444cbb0a74\") " pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:01.809753 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:01.809711    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:46:01.809753 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:01.809723    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:46:01.809753 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:01.809732    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2rlr5 for pod openshift-network-diagnostics/network-check-target-hvfh7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:46:01.809859 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:01.809770    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5 podName:e5ac8782-ecd9-4940-85a8-af444cbb0a74 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:17.809758516 +0000 UTC m=+34.303092434 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rlr5" (UniqueName: "kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5") pod "network-check-target-hvfh7" (UID: "e5ac8782-ecd9-4940-85a8-af444cbb0a74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:46:01.922872 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:01.922852    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:01.922981 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:01.922881    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:01.922981 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:01.922946    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hvfh7" podUID="e5ac8782-ecd9-4940-85a8-af444cbb0a74"
Oct 29 21:46:01.923178 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:01.923132    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t68qw" podUID="aff21346-754f-4de1-b935-26f9a51145fe"
Oct 29 21:46:02.147389 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:02.147324    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj" event={"ID":"7b9b32d7-fb57-41c2-8b64-e5e4a2449169","Type":"ContainerStarted","Data":"f78e57a269bc8d725c43c80159874e7004184dd7ff8a0363f47415fcccce3702"}
Oct 29 21:46:02.147816 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:02.147389    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj" event={"ID":"7b9b32d7-fb57-41c2-8b64-e5e4a2449169","Type":"ContainerStarted","Data":"e3d36e20e658a3eae38d6e43c20e55f78c0689fe20ccbacf519c50c35fa911c1"}
Oct 29 21:46:02.147816 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:02.147438    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj" event={"ID":"7b9b32d7-fb57-41c2-8b64-e5e4a2449169","Type":"ContainerStarted","Data":"4f123ec7fe3bdf7ff5d2dc2efeb9151e9936850cab29ed755cd43f5f45cddce8"}
Oct 29 21:46:02.150383 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:02.150361    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4kdn5" event={"ID":"bf9beffb-08c3-4766-bdd8-cdeaa557e5be","Type":"ContainerStarted","Data":"3a463c55deed0d8f9864b0ecf155e34f2d73a08cfd5a003d5d058ecba9f58715"}
Oct 29 21:46:02.203201 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:02.201951    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4kdn5" podStartSLOduration=7.775476776 podStartE2EDuration="18.201934386s" totalImagesPullingTime="10.42645761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:02.201091747 +0000 UTC m=+18.694425688" watchObservedRunningTime="2026-10-29 21:46:02.201934386 +0000 UTC m=+18.695268327"
Oct 29 21:46:02.921836 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:02.921809    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:02.921971 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:02.921901    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rvg5" podUID="9c53ce77-00c9-4509-9650-115117f93d54"
Oct 29 21:46:03.922708 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:03.922681    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:03.923250 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:03.922792    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t68qw" podUID="aff21346-754f-4de1-b935-26f9a51145fe"
Oct 29 21:46:03.923250 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:03.922846    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:03.923250 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:03.922936    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hvfh7" podUID="e5ac8782-ecd9-4940-85a8-af444cbb0a74"
Oct 29 21:46:04.158902 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:04.158867    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj" event={"ID":"7b9b32d7-fb57-41c2-8b64-e5e4a2449169","Type":"ContainerStarted","Data":"1b3b037d98d7049019df43289933df2cc0ec6437d4f6a5617cd8c832424daa41"}
Oct 29 21:46:04.922182 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:04.922136    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:04.922317 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:04.922244    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rvg5" podUID="9c53ce77-00c9-4509-9650-115117f93d54"
Oct 29 21:46:05.922388 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:05.922356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:05.922871 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:05.922491    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t68qw" podUID="aff21346-754f-4de1-b935-26f9a51145fe"
Oct 29 21:46:05.922871 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:05.922535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:05.922871 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:05.922627    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hvfh7" podUID="e5ac8782-ecd9-4940-85a8-af444cbb0a74"
Oct 29 21:46:06.169002 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:06.168972    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj" event={"ID":"7b9b32d7-fb57-41c2-8b64-e5e4a2449169","Type":"ContainerStarted","Data":"b8376e10f35255fd3308b524c5f78e8653462147cc0ec462d544e33e5f109342"}
Oct 29 21:46:06.169268 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:06.169248    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:06.169347 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:06.169323    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:06.184573 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:06.184517    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:06.184688 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:06.184614    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:06.265619 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:06.265572    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj" podStartSLOduration=7.590960892 podStartE2EDuration="22.265560138s" totalImagesPullingTime="14.674599246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:06.265221937 +0000 UTC m=+22.758555876" watchObservedRunningTime="2026-10-29 21:46:06.265560138 +0000 UTC m=+22.758894077"
Oct 29 21:46:06.922564 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:06.922527    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:06.922947 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:06.922637    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rvg5" podUID="9c53ce77-00c9-4509-9650-115117f93d54"
Oct 29 21:46:07.171389 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:07.171355    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:46:07.487388 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:07.487325    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7rvg5"]
Oct 29 21:46:07.487503 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:07.487415    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:07.487503 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:07.487486    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rvg5" podUID="9c53ce77-00c9-4509-9650-115117f93d54"
Oct 29 21:46:07.490563 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:07.490539    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hvfh7"]
Oct 29 21:46:07.490686 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:07.490620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:07.490727 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:07.490693    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hvfh7" podUID="e5ac8782-ecd9-4940-85a8-af444cbb0a74"
Oct 29 21:46:07.500511 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:07.500492    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t68qw"]
Oct 29 21:46:07.500586 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:07.500575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:07.500671 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:07.500657    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t68qw" podUID="aff21346-754f-4de1-b935-26f9a51145fe"
Oct 29 21:46:08.173263 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:08.173237    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:46:08.851533 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:08.851502    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:08.922121 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:08.922096    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:08.922121 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:08.922128    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:08.922296 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:08.922132    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:08.922296 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:08.922212    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hvfh7" podUID="e5ac8782-ecd9-4940-85a8-af444cbb0a74"
Oct 29 21:46:08.922370 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:08.922339    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t68qw" podUID="aff21346-754f-4de1-b935-26f9a51145fe"
Oct 29 21:46:08.922413 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:08.922398    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rvg5" podUID="9c53ce77-00c9-4509-9650-115117f93d54"
Oct 29 21:46:09.073630 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:09.073596    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:10.922695 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:10.922663    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:10.922695 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:10.922680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:10.923206 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:10.922676    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:10.923206 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:10.922767    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t68qw" podUID="aff21346-754f-4de1-b935-26f9a51145fe"
Oct 29 21:46:10.923206 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:10.922818    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hvfh7" podUID="e5ac8782-ecd9-4940-85a8-af444cbb0a74"
Oct 29 21:46:10.923206 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:10.922882    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rvg5" podUID="9c53ce77-00c9-4509-9650-115117f93d54"
Oct 29 21:46:11.141869 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:11.141842    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:11.142062 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:11.142048    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:46:11.152951 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:11.152927    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:11.511748 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:11.511724    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:46:11.511896 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:11.511871    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:46:11.512273 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:11.512259    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:46:11.640822 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:11.640672    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:11.859876 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:11.859842    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:11.995428 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:11.995398    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:12.749777 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.749715    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeReady"
Oct 29 21:46:12.749938 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.749811    2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 29 21:46:12.813080 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.813056    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sn7rn"]
Oct 29 21:46:12.815873 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.815854    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x5mwg"]
Oct 29 21:46:12.815985 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.815970    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sn7rn"
Oct 29 21:46:12.818432 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.818409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:12.838324 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.838303    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 21:46:12.838486 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.838475    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 21:46:12.843136 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.843118    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 21:46:12.843630 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.843612    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 21:46:12.851002 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.850984    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 21:46:12.851002 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.850999    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-p5fxq\""
Oct 29 21:46:12.852056 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.852035    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x5mwg"]
Oct 29 21:46:12.871849 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.871835    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vdqjz\""
Oct 29 21:46:12.872351 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.872335    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sn7rn"]
Oct 29 21:46:12.893123 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.893104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbdjz\" (UniqueName: \"kubernetes.io/projected/84b63148-f00f-4506-b07d-9c0c0777034c-kube-api-access-pbdjz\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:12.893191 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.893129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czn8g\" (UniqueName: \"kubernetes.io/projected/ce4f55d6-e263-446d-bb7a-884cc4f1c74c-kube-api-access-czn8g\") pod \"ingress-canary-sn7rn\" (UID: \"ce4f55d6-e263-446d-bb7a-884cc4f1c74c\") " pod="openshift-ingress-canary/ingress-canary-sn7rn"
Oct 29 21:46:12.893227 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.893218    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:12.893309 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.893294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84b63148-f00f-4506-b07d-9c0c0777034c-tmp-dir\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:12.893342 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.893318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce4f55d6-e263-446d-bb7a-884cc4f1c74c-cert\") pod \"ingress-canary-sn7rn\" (UID: \"ce4f55d6-e263-446d-bb7a-884cc4f1c74c\") " pod="openshift-ingress-canary/ingress-canary-sn7rn"
Oct 29 21:46:12.893383 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.893352    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84b63148-f00f-4506-b07d-9c0c0777034c-config-volume\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:12.922015 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.921993    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:12.922110 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.922022    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:12.922110 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.922026    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:12.944391 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.944372    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 21:46:12.944391 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.944384    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 21:46:12.947322 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.947309    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 21:46:12.947693 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.947677    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4sldf\""
Oct 29 21:46:12.947693 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.947685    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hzxgj\""
Oct 29 21:46:12.957178 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.957164    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 21:46:12.957227 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.957192    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsffp\""
Oct 29 21:46:12.994423 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.994406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:12.994478 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.994432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84b63148-f00f-4506-b07d-9c0c0777034c-tmp-dir\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:12.994478 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.994451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce4f55d6-e263-446d-bb7a-884cc4f1c74c-cert\") pod \"ingress-canary-sn7rn\" (UID: \"ce4f55d6-e263-446d-bb7a-884cc4f1c74c\") " pod="openshift-ingress-canary/ingress-canary-sn7rn"
Oct 29 21:46:12.994574 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.994478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84b63148-f00f-4506-b07d-9c0c0777034c-config-volume\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:12.994574 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.994533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbdjz\" (UniqueName: \"kubernetes.io/projected/84b63148-f00f-4506-b07d-9c0c0777034c-kube-api-access-pbdjz\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:12.994574 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.994561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czn8g\" (UniqueName: \"kubernetes.io/projected/ce4f55d6-e263-446d-bb7a-884cc4f1c74c-kube-api-access-czn8g\") pod \"ingress-canary-sn7rn\" (UID: \"ce4f55d6-e263-446d-bb7a-884cc4f1c74c\") " pod="openshift-ingress-canary/ingress-canary-sn7rn"
Oct 29 21:46:12.994713 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:12.994588    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 21:46:12.994713 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:12.994665    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls podName:84b63148-f00f-4506-b07d-9c0c0777034c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:13.494642808 +0000 UTC m=+29.987976740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls") pod "dns-default-x5mwg" (UID: "84b63148-f00f-4506-b07d-9c0c0777034c") : secret "dns-default-metrics-tls" not found
Oct 29 21:46:12.994812 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.994752    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84b63148-f00f-4506-b07d-9c0c0777034c-tmp-dir\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:12.995036 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.995012    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84b63148-f00f-4506-b07d-9c0c0777034c-config-volume\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:12.998415 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:12.998399    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce4f55d6-e263-446d-bb7a-884cc4f1c74c-cert\") pod \"ingress-canary-sn7rn\" (UID: \"ce4f55d6-e263-446d-bb7a-884cc4f1c74c\") " pod="openshift-ingress-canary/ingress-canary-sn7rn"
Oct 29 21:46:13.023270 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:13.023220    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czn8g\" (UniqueName: \"kubernetes.io/projected/ce4f55d6-e263-446d-bb7a-884cc4f1c74c-kube-api-access-czn8g\") pod \"ingress-canary-sn7rn\" (UID: \"ce4f55d6-e263-446d-bb7a-884cc4f1c74c\") " pod="openshift-ingress-canary/ingress-canary-sn7rn"
Oct 29 21:46:13.024190 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:13.024170    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbdjz\" (UniqueName: \"kubernetes.io/projected/84b63148-f00f-4506-b07d-9c0c0777034c-kube-api-access-pbdjz\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:13.128282 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:13.128260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sn7rn"
Oct 29 21:46:13.436191 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:13.435926    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sn7rn"]
Oct 29 21:46:13.439516 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:46:13.439492    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4f55d6_e263_446d_bb7a_884cc4f1c74c.slice/crio-94a990d84d550b20559475c37c1036a4f63c2cf784d73674f4b782ec8b1e14a8 WatchSource:0}: Error finding container 94a990d84d550b20559475c37c1036a4f63c2cf784d73674f4b782ec8b1e14a8: Status 404 returned error can't find the container with id 94a990d84d550b20559475c37c1036a4f63c2cf784d73674f4b782ec8b1e14a8
Oct 29 21:46:13.498027 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:13.498002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:13.498127 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:13.498103    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 21:46:13.498197 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:13.498162    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls podName:84b63148-f00f-4506-b07d-9c0c0777034c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:14.498134987 +0000 UTC m=+30.991468905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls") pod "dns-default-x5mwg" (UID: "84b63148-f00f-4506-b07d-9c0c0777034c") : secret "dns-default-metrics-tls" not found
Oct 29 21:46:14.186918 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:14.186884    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sn7rn" event={"ID":"ce4f55d6-e263-446d-bb7a-884cc4f1c74c","Type":"ContainerStarted","Data":"94a990d84d550b20559475c37c1036a4f63c2cf784d73674f4b782ec8b1e14a8"}
Oct 29 21:46:14.505176 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:14.505064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:14.505317 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:14.505265    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 21:46:14.505356 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:14.505346    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls podName:84b63148-f00f-4506-b07d-9c0c0777034c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:16.505325249 +0000 UTC m=+32.998659181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls") pod "dns-default-x5mwg" (UID: "84b63148-f00f-4506-b07d-9c0c0777034c") : secret "dns-default-metrics-tls" not found
Oct 29 21:46:14.550306 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:14.550272    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:14.591878 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:14.591105    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-j8x4s"]
Oct 29 21:46:14.613677 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:14.613629    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-j8x4s"
Oct 29 21:46:14.615314 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:14.615285    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-j8x4s"]
Oct 29 21:46:14.635680 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:14.635658    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hxxfk\""
Oct 29 21:46:14.654221 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:14.654189    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:14.654340 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:14.654268    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 29 21:46:14.682985 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:14.682959    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:14.706969 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:14.706926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q548s\" (UniqueName: \"kubernetes.io/projected/64e43900-4506-4cca-88fc-8da6c38f07ac-kube-api-access-q548s\") pod \"migrator-66df5d496b-j8x4s\" (UID: \"64e43900-4506-4cca-88fc-8da6c38f07ac\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-j8x4s"
Oct 29 21:46:14.802616 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:14.802547    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:14.807504 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:14.807469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q548s\" (UniqueName: \"kubernetes.io/projected/64e43900-4506-4cca-88fc-8da6c38f07ac-kube-api-access-q548s\") pod \"migrator-66df5d496b-j8x4s\" (UID: \"64e43900-4506-4cca-88fc-8da6c38f07ac\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-j8x4s"
Oct 29 21:46:14.834796 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:14.834769    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q548s\" (UniqueName: \"kubernetes.io/projected/64e43900-4506-4cca-88fc-8da6c38f07ac-kube-api-access-q548s\") pod \"migrator-66df5d496b-j8x4s\" (UID: \"64e43900-4506-4cca-88fc-8da6c38f07ac\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-j8x4s"
Oct 29 21:46:14.953728 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:14.953698    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-j8x4s"
Oct 29 21:46:15.123611 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.123577    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-h7nlm"]
Oct 29 21:46:15.129605 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.129580    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:15.146275 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.146216    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-h7nlm"]
Oct 29 21:46:15.148258 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.148079    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 29 21:46:15.148258 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.148081    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 29 21:46:15.148258 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.148083    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pvv78\""
Oct 29 21:46:15.148258 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.148119    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 29 21:46:15.148258 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.148102    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 29 21:46:15.211992 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.211961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/83f38264-bee5-4874-8bb0-e8ab38d21598-signing-cabundle\") pod \"service-ca-5fffc97bdf-h7nlm\" (UID: \"83f38264-bee5-4874-8bb0-e8ab38d21598\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:15.212399 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.212021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2q57\" (UniqueName: \"kubernetes.io/projected/83f38264-bee5-4874-8bb0-e8ab38d21598-kube-api-access-b2q57\") pod \"service-ca-5fffc97bdf-h7nlm\" (UID: \"83f38264-bee5-4874-8bb0-e8ab38d21598\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:15.212399 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.212221    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/83f38264-bee5-4874-8bb0-e8ab38d21598-signing-key\") pod \"service-ca-5fffc97bdf-h7nlm\" (UID: \"83f38264-bee5-4874-8bb0-e8ab38d21598\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:15.278214 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.278184    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-j8x4s"]
Oct 29 21:46:15.281336 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:46:15.281302    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e43900_4506_4cca_88fc_8da6c38f07ac.slice/crio-2c697a3308b3ce54e0c1b9bfc7076c6884fedbc1167b148c7634b0a59e462bb7 WatchSource:0}: Error finding container 2c697a3308b3ce54e0c1b9bfc7076c6884fedbc1167b148c7634b0a59e462bb7: Status 404 returned error can't find the container with id 2c697a3308b3ce54e0c1b9bfc7076c6884fedbc1167b148c7634b0a59e462bb7
Oct 29 21:46:15.313073 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.313051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2q57\" (UniqueName: \"kubernetes.io/projected/83f38264-bee5-4874-8bb0-e8ab38d21598-kube-api-access-b2q57\") pod \"service-ca-5fffc97bdf-h7nlm\" (UID: \"83f38264-bee5-4874-8bb0-e8ab38d21598\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:15.313213 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.313113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/83f38264-bee5-4874-8bb0-e8ab38d21598-signing-key\") pod \"service-ca-5fffc97bdf-h7nlm\" (UID: \"83f38264-bee5-4874-8bb0-e8ab38d21598\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:15.313213 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.313188    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/83f38264-bee5-4874-8bb0-e8ab38d21598-signing-cabundle\") pod \"service-ca-5fffc97bdf-h7nlm\" (UID: \"83f38264-bee5-4874-8bb0-e8ab38d21598\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:15.313972 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.313946    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/83f38264-bee5-4874-8bb0-e8ab38d21598-signing-cabundle\") pod \"service-ca-5fffc97bdf-h7nlm\" (UID: \"83f38264-bee5-4874-8bb0-e8ab38d21598\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:15.315656 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.315635    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/83f38264-bee5-4874-8bb0-e8ab38d21598-signing-key\") pod \"service-ca-5fffc97bdf-h7nlm\" (UID: \"83f38264-bee5-4874-8bb0-e8ab38d21598\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:15.338410 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.338388    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2q57\" (UniqueName: \"kubernetes.io/projected/83f38264-bee5-4874-8bb0-e8ab38d21598-kube-api-access-b2q57\") pod \"service-ca-5fffc97bdf-h7nlm\" (UID: \"83f38264-bee5-4874-8bb0-e8ab38d21598\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:15.451292 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.451212    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:15.522692 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.522627    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:15.545074 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.543936    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d8bd7"]
Oct 29 21:46:15.574712 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.574688    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d8bd7"]
Oct 29 21:46:15.574844 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.574815    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:15.592301 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.592273    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 21:46:15.592404 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.592326    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w9rdw\""
Oct 29 21:46:15.592810 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.592795    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 21:46:15.592913 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.592896    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 21:46:15.593082 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.593067    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 21:46:15.698936 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.698901    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:15.716936 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.716858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:15.716936 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.716930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vln5d\" (UniqueName: \"kubernetes.io/projected/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-api-access-vln5d\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:15.717112 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.717038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0bd6859-5176-430c-88cc-507c3f6c136b-data-volume\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:15.717112 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.717066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:15.717226 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.717195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0bd6859-5176-430c-88cc-507c3f6c136b-crio-socket\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:15.777409 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.777374    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-h7nlm"]
Oct 29 21:46:15.780916 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:46:15.780884    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f38264_bee5_4874_8bb0_e8ab38d21598.slice/crio-fcd162a8916ef90eb06ed9773f93b08a716748efea7d71f9a8ce9317ecf33f35 WatchSource:0}: Error finding container fcd162a8916ef90eb06ed9773f93b08a716748efea7d71f9a8ce9317ecf33f35: Status 404 returned error can't find the container with id fcd162a8916ef90eb06ed9773f93b08a716748efea7d71f9a8ce9317ecf33f35
Oct 29 21:46:15.818478 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.818427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0bd6859-5176-430c-88cc-507c3f6c136b-crio-socket\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:15.818616 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.818524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:15.818616 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.818581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vln5d\" (UniqueName: \"kubernetes.io/projected/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-api-access-vln5d\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:15.818616 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.818607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0bd6859-5176-430c-88cc-507c3f6c136b-data-volume\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:15.818764 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.818629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:15.818764 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.818648    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0bd6859-5176-430c-88cc-507c3f6c136b-crio-socket\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:15.818764 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:15.818666    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:15.818764 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:15.818729    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 21:46:15.818764 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:15.818743    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls podName:a0bd6859-5176-430c-88cc-507c3f6c136b nodeName:}" failed. No retries permitted until 2026-10-29 21:46:16.318725502 +0000 UTC m=+32.812059433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8bd7" (UID: "a0bd6859-5176-430c-88cc-507c3f6c136b") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:15.818954 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:15.818790    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm podName:a0bd6859-5176-430c-88cc-507c3f6c136b nodeName:}" failed. No retries permitted until 2026-10-29 21:46:16.318769634 +0000 UTC m=+32.812103554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d8bd7" (UID: "a0bd6859-5176-430c-88cc-507c3f6c136b") : configmap "kube-rbac-proxy" not found
Oct 29 21:46:15.818954 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.818897    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0bd6859-5176-430c-88cc-507c3f6c136b-data-volume\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:15.843588 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:15.843559    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vln5d\" (UniqueName: \"kubernetes.io/projected/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-api-access-vln5d\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:16.149806 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:16.149769    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:16.192867 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:16.192834    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-j8x4s" event={"ID":"64e43900-4506-4cca-88fc-8da6c38f07ac","Type":"ContainerStarted","Data":"2c697a3308b3ce54e0c1b9bfc7076c6884fedbc1167b148c7634b0a59e462bb7"}
Oct 29 21:46:16.193843 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:16.193813    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm" event={"ID":"83f38264-bee5-4874-8bb0-e8ab38d21598","Type":"ContainerStarted","Data":"fcd162a8916ef90eb06ed9773f93b08a716748efea7d71f9a8ce9317ecf33f35"}
Oct 29 21:46:16.321652 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:16.321616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:16.322048 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:16.321684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:16.322048 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:16.321734    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:16.322048 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:16.321765    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 21:46:16.322048 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:16.321809    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls podName:a0bd6859-5176-430c-88cc-507c3f6c136b nodeName:}" failed. No retries permitted until 2026-10-29 21:46:17.321787208 +0000 UTC m=+33.815121127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8bd7" (UID: "a0bd6859-5176-430c-88cc-507c3f6c136b") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:16.322048 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:16.321856    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm podName:a0bd6859-5176-430c-88cc-507c3f6c136b nodeName:}" failed. No retries permitted until 2026-10-29 21:46:17.321843688 +0000 UTC m=+33.815177610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d8bd7" (UID: "a0bd6859-5176-430c-88cc-507c3f6c136b") : configmap "kube-rbac-proxy" not found
Oct 29 21:46:16.523681 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:16.523609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:16.523843 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:16.523791    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 21:46:16.523898 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:16.523860    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls podName:84b63148-f00f-4506-b07d-9c0c0777034c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:20.523842637 +0000 UTC m=+37.017176578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls") pod "dns-default-x5mwg" (UID: "84b63148-f00f-4506-b07d-9c0c0777034c") : secret "dns-default-metrics-tls" not found
Oct 29 21:46:17.198954 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:17.198854    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sn7rn" event={"ID":"ce4f55d6-e263-446d-bb7a-884cc4f1c74c","Type":"ContainerStarted","Data":"79c32dfda3a09b1af5811e77980b78d1f70661c33127f8952d0bcd132c118943"}
Oct 29 21:46:17.200852 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:17.200827    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-j8x4s" event={"ID":"64e43900-4506-4cca-88fc-8da6c38f07ac","Type":"ContainerStarted","Data":"040baf6329293c34a4da0bd42602d4744bdbcee03aece79b6c23131bc22ce5c9"}
Oct 29 21:46:17.248285 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:17.248245    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sn7rn" podStartSLOduration=2.478455465 podStartE2EDuration="5.248233919s" totalImagesPullingTime="2.769778454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:17.24809358 +0000 UTC m=+33.741427520" watchObservedRunningTime="2026-10-29 21:46:17.248233919 +0000 UTC m=+33.741567855"
Oct 29 21:46:17.299269 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:17.299208    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-j8x4s" podStartSLOduration=1.7412457369999998 podStartE2EDuration="3.299192029s" totalImagesPullingTime="1.557946292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:17.298763995 +0000 UTC m=+33.792098032" watchObservedRunningTime="2026-10-29 21:46:17.299192029 +0000 UTC m=+33.792525969"
Oct 29 21:46:17.331062 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:17.331026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:17.331549 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:17.331089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:17.331549 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:17.331192    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:17.331549 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:17.331204    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 21:46:17.331549 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:17.331255    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls podName:a0bd6859-5176-430c-88cc-507c3f6c136b nodeName:}" failed. No retries permitted until 2026-10-29 21:46:19.331241784 +0000 UTC m=+35.824575741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8bd7" (UID: "a0bd6859-5176-430c-88cc-507c3f6c136b") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:17.331549 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:17.331271    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm podName:a0bd6859-5176-430c-88cc-507c3f6c136b nodeName:}" failed. No retries permitted until 2026-10-29 21:46:19.331264388 +0000 UTC m=+35.824598306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d8bd7" (UID: "a0bd6859-5176-430c-88cc-507c3f6c136b") : configmap "kube-rbac-proxy" not found
Oct 29 21:46:17.735872 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:17.735792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:17.735872 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:17.735851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:17.736083 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:17.735922    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 29 21:46:17.736083 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:17.735995    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs podName:aff21346-754f-4de1-b935-26f9a51145fe nodeName:}" failed. No retries permitted until 2026-10-29 21:46:49.735975808 +0000 UTC m=+66.229309736 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs") pod "network-metrics-daemon-t68qw" (UID: "aff21346-754f-4de1-b935-26f9a51145fe") : secret "metrics-daemon-secret" not found
Oct 29 21:46:17.738351 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:17.738328    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:17.743256 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:17.743232    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:17.836894 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:17.836866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlr5\" (UniqueName: \"kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5\") pod \"network-check-target-hvfh7\" (UID: \"e5ac8782-ecd9-4940-85a8-af444cbb0a74\") " pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:17.839535 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:17.839502    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rlr5\" (UniqueName: \"kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5\") pod \"network-check-target-hvfh7\" (UID: \"e5ac8782-ecd9-4940-85a8-af444cbb0a74\") " pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:18.038477 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:18.038396    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:18.062762 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:18.062739    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7rvg5"]
Oct 29 21:46:18.066083 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:46:18.066050    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c53ce77_00c9_4509_9650_115117f93d54.slice/crio-394dc6a3f2d75ae5fa75894c611947fb11bff76f9c1d8623cf765704782349ab WatchSource:0}: Error finding container 394dc6a3f2d75ae5fa75894c611947fb11bff76f9c1d8623cf765704782349ab: Status 404 returned error can't find the container with id 394dc6a3f2d75ae5fa75894c611947fb11bff76f9c1d8623cf765704782349ab
Oct 29 21:46:18.204468 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:18.204434    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7rvg5" event={"ID":"9c53ce77-00c9-4509-9650-115117f93d54","Type":"ContainerStarted","Data":"394dc6a3f2d75ae5fa75894c611947fb11bff76f9c1d8623cf765704782349ab"}
Oct 29 21:46:18.363051 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:18.363024    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hvfh7"]
Oct 29 21:46:18.365587 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:46:18.365560    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ac8782_ecd9_4940_85a8_af444cbb0a74.slice/crio-d3ca4c9fa5be46178328391879908ddc82677da6479545de140ba6f9756861d4 WatchSource:0}: Error finding container d3ca4c9fa5be46178328391879908ddc82677da6479545de140ba6f9756861d4: Status 404 returned error can't find the container with id d3ca4c9fa5be46178328391879908ddc82677da6479545de140ba6f9756861d4
Oct 29 21:46:18.749959 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:18.749921    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:19.209437 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:19.209401    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hvfh7" event={"ID":"e5ac8782-ecd9-4940-85a8-af444cbb0a74","Type":"ContainerStarted","Data":"d3ca4c9fa5be46178328391879908ddc82677da6479545de140ba6f9756861d4"}
Oct 29 21:46:19.211685 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:19.211655    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm" event={"ID":"83f38264-bee5-4874-8bb0-e8ab38d21598","Type":"ContainerStarted","Data":"ce9d47f72ce81d8525e4795efcbd08be5345dc7b10e856d64d5e5bf07a581842"}
Oct 29 21:46:19.348155 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:19.348115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:19.348319 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:19.348227    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:19.349004 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:19.348978    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:19.349109 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:19.349047    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls podName:a0bd6859-5176-430c-88cc-507c3f6c136b nodeName:}" failed. No retries permitted until 2026-10-29 21:46:23.34902792 +0000 UTC m=+39.842361852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8bd7" (UID: "a0bd6859-5176-430c-88cc-507c3f6c136b") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:19.349583 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:19.349561    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 21:46:19.349676 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:19.349616    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm podName:a0bd6859-5176-430c-88cc-507c3f6c136b nodeName:}" failed. No retries permitted until 2026-10-29 21:46:23.34959798 +0000 UTC m=+39.842931912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d8bd7" (UID: "a0bd6859-5176-430c-88cc-507c3f6c136b") : configmap "kube-rbac-proxy" not found
Oct 29 21:46:19.556870 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:19.556771    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:19.721208 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:19.721072    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:19.848513 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:19.848476    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:19.965537 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:19.965438    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:20.110374 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:20.110251    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:20.566206 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:20.566120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:20.568911 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:20.568884    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:20.634943 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:20.634912    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:20.655816 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:20.655787    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:20.783221 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:20.783169    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:20.972421 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:20.971708    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm" podStartSLOduration=3.149776668 podStartE2EDuration="5.971687613s" totalImagesPullingTime="2.821910945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:19.254796609 +0000 UTC m=+35.748130566" watchObservedRunningTime="2026-10-29 21:46:20.971687613 +0000 UTC m=+37.465021553"
Oct 29 21:46:20.973342 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:20.973233    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x5mwg"]
Oct 29 21:46:20.977939 ip-10-0-128-85 kubenswrapper[2581]: W1029 21:46:20.977902    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b63148_f00f_4506_b07d_9c0c0777034c.slice/crio-f0c8b4a7fa3357da053fb5546752e20e11dc25da656a8c87ed530598d5cfce4d WatchSource:0}: Error finding container f0c8b4a7fa3357da053fb5546752e20e11dc25da656a8c87ed530598d5cfce4d: Status 404 returned error can't find the container with id f0c8b4a7fa3357da053fb5546752e20e11dc25da656a8c87ed530598d5cfce4d
Oct 29 21:46:21.222684 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:21.222609    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x5mwg" event={"ID":"84b63148-f00f-4506-b07d-9c0c0777034c","Type":"ContainerStarted","Data":"f0c8b4a7fa3357da053fb5546752e20e11dc25da656a8c87ed530598d5cfce4d"}
Oct 29 21:46:21.291060 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:21.291017    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:21.501913 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:21.501831    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:21.688867 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:21.688831    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:21.898116 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:21.898084    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:22.088397 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:22.088357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:22.228183 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:22.228089    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hvfh7" event={"ID":"e5ac8782-ecd9-4940-85a8-af444cbb0a74","Type":"ContainerStarted","Data":"2407d80d656ed30958fcc1109d9e2d8bca710aca32e982a57d8f474bdc464ccb"}
Oct 29 21:46:22.240562 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:22.238261    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:22.281843 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:22.281801    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hvfh7" podStartSLOduration=34.646957937 podStartE2EDuration="38.281789389s" totalImagesPullingTime="3.634831452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:22.281332982 +0000 UTC m=+38.774666933" watchObservedRunningTime="2026-10-29 21:46:22.281789389 +0000 UTC m=+38.775123328"
Oct 29 21:46:23.292743 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:23.292704    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:23.391648 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:23.391625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:23.391815 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:23.391659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:23.391815 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:23.391759    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:23.391936 ip-10-0-128-85 kubenswrapper[2581]: E1029 21:46:23.391825    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls podName:a0bd6859-5176-430c-88cc-507c3f6c136b nodeName:}" failed. No retries permitted until 2026-10-29 21:46:31.391807997 +0000 UTC m=+47.885141933 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8bd7" (UID: "a0bd6859-5176-430c-88cc-507c3f6c136b") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:23.392037 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:23.392022    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:23.511939 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:23.511914    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:23.735266 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:23.735231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:23.791695 ip-10-0-128-85 kubenswrapper[2581]: I1029 21:46:23.791629    2581 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 21:46:23.791820 ip-10-0-128-85 systemd[1]: Stopping Kubernetes Kubelet...
Oct 29 21:46:23.803502 ip-10-0-128-85 systemd[1]: kubelet.service: Deactivated successfully.
Oct 29 21:46:23.803715 ip-10-0-128-85 systemd[1]: Stopped Kubernetes Kubelet.
Oct 29 21:46:23.803904 ip-10-0-128-85 systemd[1]: kubelet.service: Consumed 2.378s CPU time.
Oct 29 21:46:23.812884 ip-10-0-128-85 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 21:46:23.899072 ip-10-0-128-85 kubenswrapper[4550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:46:23.899072 ip-10-0-128-85 kubenswrapper[4550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 21:46:23.899072 ip-10-0-128-85 kubenswrapper[4550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:46:23.899072 ip-10-0-128-85 kubenswrapper[4550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:46:23.901169 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901083    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:23.901169 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901168    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:23.901169 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901172    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901175    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901178    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901181    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901183    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901186    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901188    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901191    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901194    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901196    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901200    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901203    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901205    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901208    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901210    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901217    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901220    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901223    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901227    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:23.901267 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901231    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901235    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901239    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901242    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901244    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901247    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901250    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901252    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901255    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901258    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901260    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901264    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901267    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901270    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901273    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901276    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901288    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901291    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901294    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:23.901777 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901297    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901299    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901302    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901304    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901307    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901311    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901315    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901318    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901321    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901324    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901328    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901330    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901333    4550 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901336    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901339    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901341    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901344    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901347    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901349    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:23.902270 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901352    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901355    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901357    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901360    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901362    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901366    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901368    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901371    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901374    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901376    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901379    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901381    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901384    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901387    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901390    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901392    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901395    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901398    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901400    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901403    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:23.902733 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901405    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901408    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901411    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901413    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901417    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901419    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901422    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901424    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901427    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901429    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901432    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901434    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901437    4550 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901439    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901442    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901445    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901447    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901450    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901453    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901455    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:23.903250 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901458    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901461    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901463    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901467    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901472    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901475    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.901477    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902286    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902295    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902299    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902302    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902306    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902308    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902311    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902314    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902316    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902319    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902323    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902325    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:23.903741 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902328    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902331    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902333    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902336    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902338    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902341    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902343    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902346    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902348    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902351    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902354    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902357    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902359    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902362    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902364    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902367    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902369    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902373    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902376    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902378    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:23.904215 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902380    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902383    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902386    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902388    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902391    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902393    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902395    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902398    4550 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902401    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902403    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902406    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902409    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902411    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902414    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902416    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902419    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902421    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902424    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902427    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902429    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:23.904715 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902432    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902434    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902438    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902440    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902443    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902446    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902448    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902450    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902453    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902455    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902458    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902460    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902463    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902465    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902468    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902470    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902473    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902475    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902478    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:23.905233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902480    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902483    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902485    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902488    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902491    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902495    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902498    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902500    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902503    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902505    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902508    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902510    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902513    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902516    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902518    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902521    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902524    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902527    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902529    4550 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902532    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:23.905689 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902534    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902537    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902539    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902542    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902545    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902549    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902552    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902556    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902559    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902562    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902564    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902567    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902569    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902572    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.902574    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903063    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903070    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903075    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903078    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:23.906211 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903081    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903084    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903087    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903090    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903093    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903098    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903101    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903103    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903106    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903109    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903111    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903114    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903116    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903119    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903121    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903124    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903127    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903129    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903132    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903134    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:23.906699 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903137    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903155    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903159    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903163    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903166    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903169    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903171    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903174    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903177    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903180    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903182    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903185    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903188    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903191    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903194    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903196    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903201    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903205    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903207    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:23.907255 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903210    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903213    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903216    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903218    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903221    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903223    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903227    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903231    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903234    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903237    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903240    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903243    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903246    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903248    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903251    4550 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903254    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903256    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903259    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903262    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:23.907728 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903264    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903267    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903269    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903272    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903274    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903276    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903280    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903282    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903285    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903287    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903295    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903299    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903302    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903305    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903307    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903310    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903312    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903315    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903318    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903320    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:23.908195 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903323    4550 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903325    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903328    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903330    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903333    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903335    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903338    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903340    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903343    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903345    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903348    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903350    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903353    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903356    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903358    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903360    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903363    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903365    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903368    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:23.910013 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903371    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903374    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903376    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903381    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.903385    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903476    4550 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903487    4550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903497    4550 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903501    4550 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903506    4550 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903509    4550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903513    4550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903517    4550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903521    4550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903524    4550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903527    4550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903530    4550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903534    4550 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903537    4550 flags.go:64] FLAG: --cgroup-root=""
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903540    4550 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903544    4550 flags.go:64] FLAG: --client-ca-file=""
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903547    4550 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903549    4550 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903554    4550 flags.go:64] FLAG: --cluster-domain=""
Oct 29 21:46:23.910516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903557    4550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903560    4550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903563    4550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903567    4550 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903574    4550 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903577    4550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903580    4550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903584    4550 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903587    4550 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903590    4550 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903593    4550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903599    4550 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903603    4550 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903607    4550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903610    4550 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903613    4550 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903616    4550 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903620    4550 flags.go:64] FLAG: --enable-server="true"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903623    4550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903627    4550 flags.go:64] FLAG: --event-burst="100"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903630    4550 flags.go:64] FLAG: --event-qps="50"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903633    4550 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903636    4550 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903639    4550 flags.go:64] FLAG: --eviction-hard=""
Oct 29 21:46:23.911352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903643    4550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903646    4550 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903649    4550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903652    4550 flags.go:64] FLAG: --eviction-soft=""
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903655    4550 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903658    4550 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903661    4550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903664    4550 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903667    4550 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903670    4550 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903673    4550 flags.go:64] FLAG: --feature-gates=""
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903677    4550 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903680    4550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903683    4550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903687    4550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903690    4550 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903693    4550 flags.go:64] FLAG: --help="false"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903696    4550 flags.go:64] FLAG: --hostname-override="ip-10-0-128-85.ec2.internal"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903700    4550 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903704    4550 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903708    4550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903712    4550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903715    4550 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903718    4550 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 21:46:23.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903721    4550 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903724    4550 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903727    4550 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903730    4550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903733    4550 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903736    4550 flags.go:64] FLAG: --kube-reserved=""
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903739    4550 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903742    4550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903745    4550 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903748    4550 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903751    4550 flags.go:64] FLAG: --lock-file=""
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903754    4550 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903757    4550 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903760    4550 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903765    4550 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903768    4550 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903771    4550 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903774    4550 flags.go:64] FLAG: --logging-format="text"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903777    4550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903780    4550 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903783    4550 flags.go:64] FLAG: --manifest-url=""
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903786    4550 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903790    4550 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903794    4550 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903797    4550 flags.go:64] FLAG: --max-pods="110"
Oct 29 21:46:23.912939 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903800    4550 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903803    4550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903808    4550 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903813    4550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903816    4550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903819    4550 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903822    4550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903828    4550 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903831    4550 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903834    4550 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903838    4550 flags.go:64] FLAG: --pod-cidr=""
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903840    4550 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903843    4550 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903846    4550 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903849    4550 flags.go:64] FLAG: --port="10250"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903852    4550 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903855    4550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-003a887b235a32254"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903858    4550 flags.go:64] FLAG: --qos-reserved=""
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903861    4550 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903864    4550 flags.go:64] FLAG: --register-node="true"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903867    4550 flags.go:64] FLAG: --register-with-taints=""
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903871    4550 flags.go:64] FLAG: --registry-burst="10"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903874    4550 flags.go:64] FLAG: --registry-qps="5"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903877    4550 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903880    4550 flags.go:64] FLAG: --reserved-memory=""
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903883    4550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 21:46:23.913583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903886    4550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903889    4550 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903892    4550 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903895    4550 flags.go:64] FLAG: --runonce="false"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903898    4550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903901    4550 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903904    4550 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903907    4550 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903911    4550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903916    4550 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903920    4550 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903923    4550 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903926    4550 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903929    4550 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903932    4550 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903934    4550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903937    4550 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903940    4550 flags.go:64] FLAG: --system-cgroups=""
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903943    4550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903949    4550 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903952    4550 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903955    4550 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903959    4550 flags.go:64] FLAG: --tls-min-version=""
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903962    4550 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903965    4550 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 21:46:23.914230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903968    4550 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903971    4550 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903974    4550 flags.go:64] FLAG: --v="2"
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903978    4550 flags.go:64] FLAG: --version="false"
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903982    4550 flags.go:64] FLAG: --vmodule=""
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903986    4550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.903990    4550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904130    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904134    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904151    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904157    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904161    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904165    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904168    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904171    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904175    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904179    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904183    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904186    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904188    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904191    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:23.914851 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904193    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904196    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904199    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904201    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904204    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904206    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904209    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904212    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904215    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904218    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904221    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904223    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904226    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904230    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904234    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904238    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904241    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904244    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904246    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:23.915609 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904249    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904252    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904254    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904257    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904259    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904262    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904264    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904269    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904272    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904275    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904278    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904281    4550 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904283    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904286    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904288    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904291    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904293    4550 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904296    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904299    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904301    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:23.916113 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904304    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904306    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904309    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904311    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904314    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904317    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904319    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904322    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904324    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904327    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904329    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904332    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904334    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904337    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904340    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904344    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904347    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904350    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904353    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904357    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:23.916704 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904359    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904362    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904365    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904368    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904370    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904373    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904375    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904378    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904381    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904383    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904386    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904388    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904391    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904394    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904396    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904399    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904401    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904404    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904406    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:23.917222 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904409    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904411    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904414    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904416    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904419    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904421    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904424    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904426    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904429    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904431    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904434    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904436    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904440    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904442    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.904468    4550 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904509    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904513    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:23.917737 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904517    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904520    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904523    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904526    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904528    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904531    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904533    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904536    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904538    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904541    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904544    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904546    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904549    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904551    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904554    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904556    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904559    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904562    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904564    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:23.918320 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904567    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904569    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904572    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904574    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904577    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904579    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904582    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904592    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904596    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904599    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904601    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904604    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904606    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904609    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904612    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904615    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904618    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904620    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904623    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904625    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:23.918905 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904628    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904631    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904633    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904636    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904638    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904641    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904643    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904646    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904648    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904651    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904653    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904656    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904658    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904661    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904663    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904666    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904668    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904671    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904675    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:23.919753 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904677    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904681    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904684    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904688    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904692    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904695    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904698    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904701    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904704    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904707    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904709    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904712    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904715    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904717    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904720    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904723    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904726    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904729    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904731    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904734    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:23.920559 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904736    4550 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904740    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904743    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904745    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904747    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904750    4550 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904752    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904755    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904758    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904760    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904763    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904767    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904769    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904773    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904776    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904779    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904781    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904784    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904787    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904789    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:23.921546 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904792    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904795    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904798    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904802    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904806    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.904809    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.908436    4550 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.908452    4550 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.908472    4550 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.908476    4550 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908602    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908608    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908611    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908615    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908619    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908622    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908630    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908633    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908636    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:23.922402 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908639    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908642    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908645    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908648    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908651    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908654    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908657    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908660    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908662    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908678    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908681    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908684    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908687    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908689    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908692    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908695    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908698    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908701    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908704    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908706    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:23.923257 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908709    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908712    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908717    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908720    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908722    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908725    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908728    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908731    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908733    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908736    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908739    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908742    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908745    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908748    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908751    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908756    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908759    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908761    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:23.924115 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908764    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908768    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908772    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908775    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908780    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908783    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908786    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908789    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908791    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908795    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908800    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908803    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908806    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908808    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908811    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908814    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908816    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908819    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908823    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:23.924902 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908827    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908831    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908834    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908840    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908843    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908846    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908848    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908851    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908854    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908856    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908859    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908862    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908864    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908867    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908870    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908875    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908878    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908881    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908884    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908887    4550 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:23.925842 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908890    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908893    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908896    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908899    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908902    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908904    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908907    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908909    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908915    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908918    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908921    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908924    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908926    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908929    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908931    4550 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908934    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908937    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908940    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908942    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:23.926743 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.908945    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.908979    4550 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909279    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909285    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909289    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909292    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909294    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909297    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909303    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909306    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909309    4550 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909312    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909316    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909318    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909321    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909324    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:23.927655 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909327    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909330    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909333    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909335    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909338    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909343    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909357    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909362    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909366    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909371    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909391    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909398    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909402    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909407    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909411    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909415    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909419    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909429    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909433    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909437    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:23.928403 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909442    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909446    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909451    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909455    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909459    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909463    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909468    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909472    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909477    4550 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909486    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909491    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909496    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909500    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909505    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909509    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909513    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909562    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909686    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909694    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909700    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:23.929992 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909705    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909710    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909714    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909719    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909724    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909731    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909737    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909741    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909746    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909751    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909755    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909760    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909764    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909771    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909776    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909781    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909785    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909790    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909794    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:23.930535 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909798    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909802    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909806    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909811    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909815    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909819    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909824    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909828    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909832    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909837    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909841    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909845    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909849    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909853    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909860    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909865    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909869    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909874    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909878    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909882    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:23.931214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909887    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:23.932017 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909891    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:23.932017 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909898    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:23.932017 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909902    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:23.932017 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909906    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:23.932017 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909913    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:23.932017 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909919    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:23.932017 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909923    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:23.932017 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909926    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:23.932017 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909929    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:23.932017 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909932    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:23.932017 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909936    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:23.932017 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.909939    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:23.932017 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.909975    4550 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:46:23.932017 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910013    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:23.932017 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910017    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910021    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910024    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910027    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910030    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910033    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910035    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910038    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910041    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910044    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910046    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910049    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910052    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910055    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910057    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910060    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910062    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910065    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910068    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910070    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:23.932602 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910073    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910075    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910078    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910080    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910085    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910088    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910092    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910095    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910098    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910102    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910105    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910108    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910110    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910113    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910116    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910119    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910122    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910124    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910127    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:23.933374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910129    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910132    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910135    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910159    4550 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910163    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910165    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910168    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910171    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910173    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910176    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910178    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910181    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910184    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910186    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910189    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910191    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910194    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910199    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910202    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910204    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:23.933988 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910207    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910209    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910212    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910216    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910219    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910221    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910224    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910227    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910229    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910232    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910235    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910237    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910240    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910243    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910245    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910248    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910250    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910253    4550 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910255    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910258    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:23.934508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910260    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910263    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910265    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910268    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910270    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910273    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910276    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910278    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910281    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910284    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910287    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910289    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910292    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910295    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910298    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910302    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910305    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910308    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910310    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:23.934979 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910313    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910317    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910320    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910322    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910325    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910328    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:23.910330    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.910388    4550 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.911363    4550 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.911429    4550 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.911679    4550 server.go:1014] "Starting client certificate rotation"
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.911818    4550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.911865    4550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 21:40:44 +0000 UTC" deadline="2028-05-17 02:23:20.980385864 +0000 UTC"
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.911902    4550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13564h36m57.06848542s"
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.912456    4550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.914106    4550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 21:46:23.935472 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.915796    4550 log.go:25] "Validated CRI v1 runtime API"
Oct 29 21:46:23.935862 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.917565    4550 log.go:25] "Validated CRI v1 image API"
Oct 29 21:46:23.935862 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.918326    4550 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 21:46:23.935862 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.920563    4550 fs.go:135] Filesystem UUIDs: map[22cb7a11-0711-4f85-925b-b6d85fc76f81:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d84f7fcd-5264-4fc0-ba4f-1c59059be07a:/dev/nvme0n1p4]
Oct 29 21:46:23.935950 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.920606    4550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d8115cff4151270a2a340c4c815e4edfa493fd575ee25208b986bad1f29f160/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d8115cff4151270a2a340c4c815e4edfa493fd575ee25208b986bad1f29f160/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c697a3308b3ce54e0c1b9bfc7076c6884fedbc1167b148c7634b0a59e462bb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c697a3308b3ce54e0c1b9bfc7076c6884fedbc1167b148c7634b0a59e462bb7/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/358fae070de9e341c80a1250438af79e63ec31b586d6cc2dff1073152278cba1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/358fae070de9e341c80a1250438af79e63ec31b586d6cc2dff1073152278cba1/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/394dc6a3f2d75ae5fa75894c611947fb11bff76f9c1d8623cf765704782349ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/394dc6a3f2d75ae5fa75894c611947fb11bff76f9c1d8623cf765704782349ab/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/533f521b860f11ff4902df67f658c3bd6cdf11671160b32dee2732a566477840/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/533f521b860f11ff4902df67f658c3bd6cdf11671160b32dee2732a566477840/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94a990d84d550b20559475c37c1036a4f63c2cf784d73674f4b782ec8b1e14a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94a990d84d550b20559475c37c1036a4f63c2cf784d73674f4b782ec8b1e14a8/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd77f956d4d408cdef442b8359f3a857b52b91945cbbd8648a6696c3a944d73b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd77f956d4d408cdef442b8359f3a857b52b91945cbbd8648a6696c3a944d73b/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4b2ec9adbb917d9205f405a968e2677147436e0f71b66e5e17b1a27a7c548b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4b2ec9adbb917d9205f405a968e2677147436e0f71b66e5e17b1a27a7c548b4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb7e36d875c48455585d287c440fe02e2ec49a2726b9c9de68581ba051e4445c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb7e36d875c48455585d287c440fe02e2ec49a2726b9c9de68581ba051e4445c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc38350aca267c87ffdd73d6f139252979c0d21a3017070c4ed216d04ba790db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc38350aca267c87ffdd73d6f139252979c0d21a3017070c4ed216d04ba790db/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3ca4c9fa5be46178328391879908ddc82677da6479545de140ba6f9756861d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3ca4c9fa5be46178328391879908ddc82677da6479545de140ba6f9756861d4/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da2aa0ce7d11307030dcc28b4bcfdbf08454aabdc358d41f0aa2bdf913d8df92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da2aa0ce7d11307030dcc28b4bcfdbf08454aabdc358d41f0aa2bdf913d8df92/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6dd64c8ab1801f80a647e9603e93cf037daa5e56a3626056c0b2e9a8fdffc03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6dd64c8ab1801f80a647e9603e93cf037daa5e56a3626056c0b2e9a8fdffc03/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb90ff60287a8c76afe91c2493cd641894c8a83163328042c822928d2f886ff4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb90ff60287a8c76afe91c2493cd641894c8a83163328042c822928d2f886ff4/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0c8b4a7fa3357da053fb5546752e20e11dc25da656a8c87ed530598d5cfce4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0c8b4a7fa3357da053fb5546752e20e11dc25da656a8c87ed530598d5cfce4d/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcd162a8916ef90eb06ed9773f93b08a716748efea7d71f9a8ce9317ecf33f35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcd162a8916ef90eb06ed9773f93b08a716748efea7d71f9a8ce9317ecf33f35/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09d024f0-dc43-4c99-81dd-ec9a2c126e86/volumes/kubernetes.io~projected/kube-api-access-bsk5z:{mountpoint:/var/lib/kubelet/pods/09d024f0-dc43-4c99-81dd-ec9a2c126e86/volumes/kubernetes.io~projected/kube-api-access-bsk5z major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12cd2b00-faff-422b-a588-c07050ab96f7/volumes/kubernetes.io~projected/kube-api-access-rflhg:{mountpoint:/var/lib/kubelet/pods/12cd2b00-faff-422b-a588-c07050ab96f7/volumes/kubernetes.io~projected/kube-api-access-rflhg major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/193ad4d9-4ef2-4ede-abd6-6ab5b589321a/volumes/kubernetes.io~projected/kube-api-access-ppstg:{mountpoint:/var/lib/kubelet/pods/193ad4d9-4ef2-4ede-abd6-6ab5b589321a/volumes/kubernetes.io~projected/kube-api-access-ppstg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64e43900-4506-4cca-88fc-8da6c38f07ac/volumes/kubernetes.io~projected/kube-api-access-q548s:{mountpoint:/var/lib/kubelet/pods/64e43900-4506-4cca-88fc-8da6c38f07ac/volumes/kubernetes.io~projected/kube-api-access-q548s major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9b32d7-fb57-41c2-8b64-e5e4a2449169/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7b9b32d7-fb57-41c2-8b64-e5e4a2449169/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9b32d7-fb57-41c2-8b64-e5e4a2449169/volumes/kubernetes.io~projected/kube-api-access-kwqrl:{mountpoint:/var/lib/kubelet/pods/7b9b32d7-fb57-41c2-8b64-e5e4a2449169/volumes/kubernetes.io~projected/kube-api-access-kwqrl major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b9b32d7-fb57-41c2-8b64-e5e4a2449169/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7b9b32d7-fb57-41c2-8b64-e5e4a2449169/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83f38264-bee5-4874-8bb0-e8ab38d21598/volumes/kubernetes.io~projected/kube-api-access-b2q57:{mountpoint:/var/lib/kubelet/pods/83f38264-bee5-4874-8bb0-e8ab38d21598/volumes/kubernetes.io~projected/kube-api-access-b2q57 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83f38264-bee5-4874-8bb0-e8ab38d21598/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/83f38264-bee5-4874-8bb0-e8ab38d21598/volumes/kubernetes.io~secret/signing-key major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84b63148-f00f-4506-b07d-9c0c0777034c/volumes/kubernetes.io~projected/kube-api-access-pbdjz:{mountpoint:/var/lib/kubelet/pods/84b63148-f00f-4506-b07d-9c0c0777034c/volumes/kubernetes.io~projected/kube-api-access-pbdjz major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84b63148-f00f-4506-b07d-9c0c0777034c/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/84b63148-f00f-4506-b07d-9c0c0777034c/volumes/kubernetes.io~secret/metrics-tls major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/991476c7-c67c-436a-83e0-e1514350d019/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/991476c7-c67c-436a-83e0-e1514350d019/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/991476c7-c67c-436a-83e0-e1514350d019/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/991476c7-c67c-436a-83e0-e1514350d019/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/991476c7-c67c-436a-83e0-e1514350d019/volumes/kubernetes.io~projected/kube-api-access-jm6g9:{mountpoint:/var/lib/kubelet/pods/991476c7-c67c-436a-83e0-e1514350d019/volumes/kubernetes.io~projected/kube-api-access-jm6g9 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c53ce77-00c9-4509-9650-115117f93d54/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9c53ce77-00c9-4509-9650-115117f93d54/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0bd6859-5176-430c-88cc-507c3f6c136b/volumes/kubernetes.io~projected/kube-api-access-vln5d:{mountpoint:/var/lib/kubelet/pods/a0bd6859-5176-430c-88cc-507c3f6c136b/volumes/kubernetes.io~projected/kube-api-access-vln5d major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2/volumes/kubernetes.io~projected/kube-api-access-scdz6:{mountpoint:/var/lib/kubelet/pods/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2/volumes/kubernetes.io~projected/kube-api-access-scdz6 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aff21346-754f-4de1-b935-26f9a51145fe/volumes/kubernetes.io~projected/kube-api-access-n8xqx:{mountpoint:/var/lib/kubelet/pods/aff21346-754f-4de1-b935-26f9a51145fe/volumes/kubernetes.io~projected/kube-api-access-n8xqx major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b49ef0f6-2ac3-4393-b45a-5356110e13a3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b49ef0f6-2ac3-4393-b45a-5356110e13a3/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf9beffb-08c3-4766-bdd8-cdeaa557e5be/volumes/kubernetes.io~projected/kube-api-access-z4w7s:{mountpoint:/var/lib/kubelet/pods/bf9beffb-08c3-4766-bdd8-cdeaa557e5be/volumes/kubernetes.io~projected/kube-api-access-z4w7s major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4f55d6-e263-446d-bb7a-884cc4f1c74c/volumes/kubernetes.io~projected/kube-api-access-czn8g:{mountpoint:/var/lib/kubelet/pods/ce4f55d6-e263-446d-bb7a-884cc4f1c74c/volumes/kubernetes.io~projected/kube-api-access-czn8g major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4f55d6-e263-446d-bb7a-884cc4f1c74c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/ce4f55d6-e263-446d-bb7a-884cc4f1c74c/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3d1ef12-c282-4cba-9678-e369daacc3d1/volumes/kubernetes.io~projected/kube-api-access-mjrdf:{mountpoint:/var/lib/kubelet/pods/e3d1ef12-c282-4cba-9678-e369daacc3d1/volumes/kubernetes.io~projected/kube-api-access-mjrdf major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5ac8782-ecd9-4940-85a8-af444cbb0a74/volumes/kubernetes.io~projected/kube-api-access-2rlr5:{mountpoint:/var/lib/kubelet/pods/e5ac8782-ecd9-4940-85a8-af444cbb0a74/volumes/kubernetes.io~projected/kube-api-access-2rlr5 major:0 minor:233 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/fc1fecb4a2fbb737800014ccaf4ff9761bbe604f0a5899640ed1061f3fc93d96/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a6d270fcebf0e53dda0983e3fe936dd8084df86eaa6295ce44de6eb984f1d0af/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f405c0cd9491124071a231af3426b4f5cbfd4f36e06a55d7b4dcd35c60e8f1ff/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/fcb91910b9e66d7dc0ec5b5f49eca22cbbd4a6f02521504e0ea9675fa59cb28d/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b58302ed2e16c42c6b79f4855bb725984860f9a0c1774e1480865ab5003ae3b9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e261361c68238e74d5ae038b5c15031ab7d4e1d84d781eb8826ffd65d517c43f/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f1d5da1d06306fbd811f7f95a9f24f534d7fa5b4f2e7ee53900788c2d8dd4bd1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9a13afe25a83bd2cef8f489ad2a59a3a4363691ac519a79805c9307eb8737e01/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/9caee3a77454929d984d7d257553e13fd288c67958fa7200123700a832bc5c5d/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/b82707fdc80b3b932e845bf780d1dd2070f6a75f564a499196c0f9da2a01122a/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/e5c78017868dca426c0919c9968877317c4ceea68577bffe7aa0f655182894ae/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/67d24558e0950bce7c3418eb7dd0fbb89760b889ea21606400413108705d5c0a/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/81262add07cfeb923b5e99d3748368d7da8a395e569d7831305423783283ff09/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/3554d0a7a54d66d3ad8f2832b7a804acf04c8bb3e7d19476155bde330657b673/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/019e6d45f3d664ad4bc665561d013bbd62667e26d756484548fa3e116ecca11a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/da45bc9c1e471e77dc9bcdeb3b3215b6613d03f33744630b2d24731b6d17e14b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b6af76ca66731e22f01c83b0b13c73647b8528104a678edb5fe4d49de85378e9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5e7aa027388fbba1df71a57b427507d45bf0e9c80277c69b7ef00cc5ee2d52c7/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/3b8fe5372959f838bd983d0d30ff76ca1743e682616ef6a25696d4159315018f/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-188:{mountpoint:/var/lib/containers/storage/overlay/81abfdf7f6a3cc45704ff238290bb5faaff45d260e16d20500571ed51b3f20d0/merged major:0 minor:188 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ac808e32c137590ed1e4c5f9c14e5d17afde6f2dcd8814eacc7c3ba160452701/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/0f14e90ee119845fd3a8b3611edc3b406e316de9ebe34ea954e62de23ae3f822/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/15ad1b4e1b293861ecb7e01b3b160a7c8494df04e1421582be9285eb6570bedf/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/26f93c12c305f8509433edddef0390cc02dfd0fb35a67d262f7de9d7f37a3e17/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/ed075ac5e483f9dbef02fc8df3b1c146bdcbf78a5ee42e5a9bc41c21d281d52c/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/dfe92a873ab4277a65567ff5b366edaa705ed61585d502a8ee23d8918e733d5f/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/88956a209e337b440ef7d779883852607f5eda8daf7abb7fce6cd6c3dcc8be80/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/20073dbc4204d0befefe001a51deb6c39cb2d3161809fb7b22f143cec10a40ba/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/a4298f4e46a1221e7d1412733047b33fa7f230ecf5dd6a902335473931933f92/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/e357e5488de319f158a5846f4ae9f913b3b9e8bab9ab2482ffd1a8361368998e/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/204ed35b481dc87aac448ba2b06d0c724505c5c96effa55e752d975b196c94c6/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/68290f6431118ddd35f92c2d02f7487e64164c1dd00ab68e2498a16bc65178c5/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/6c7662750b7cbcef97e6049c831507c75c1a59f1c34cf596d269ebf1ef5386ec/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/8c61e6ef2fa25e4cb0866abb74fbcfdaeaf8d20471731f06ec3b0bb8f23083f9/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/96fe92a44337a72de81aece0af6a736ed424675b624fc4e3e743b218773c7f68/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/997808bf8fb91d13ae3206a9861f3d3dab47c3465157a19e1d53c5c388fa697f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/433361ffc876ea3c5fe8f21035c3d78eae1c727d954f32aa76ac7d46eeaf0c5a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3f725c24ea43a903421b112e976a3ecf5efcbdfb0e83058c592630898e922834/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/7d442b93031abf6c288f2ae4f190adb7265d67894507707fe2126e6ac5d156cb/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/b2b64ebd866b159ae880364aefcc0b67a0c116ff6387a1a24f695f7dfd69e1e1/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/b7670565e07db69b08d690fd5e20b6045f13cb1c62579ceccf69cbbd10a77da9/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/1a373654135302f9187ec067d5fbb3de6e4c3af8ff8c85b7bef754b65368368b/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/664af2b11ba301544238851a37615ad5311010859707896182b81fcce6825df7/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9599804ecfd6e8480bbf3936e0998fa430a0f75371e1b77a3a6dfea38c5a7ea3/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.930285    4550 manager.go:217] Machine: {Timestamp:2026-10-29 21:46:23.929535406 +0000 UTC m=+0.086660898 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100260 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bdd9549a8b7fce7274c90b34d505d SystemUUID:ec2bdd95-49a8-b7fc-e727-4c90b34d505d BootID:e46ec419-cfbb-4ee3-8e3d-f71bbcb4b970 Filesystems:[{Device:/run/containers/storage/overlay-containers/d3ca4c9fa5be46178328391879908ddc82677da6479545de140ba6f9756861d4/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/358fae070de9e341c80a1250438af79e63ec31b586d6cc2dff1073152278cba1/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/84b63148-f00f-4506-b07d-9c0c0777034c/volumes/kubernetes.io~projected/kube-api-access-pbdjz DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0c8b4a7fa3357da053fb5546752e20e11dc25da656a8c87ed530598d5cfce4d/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84b63148-f00f-4506-b07d-9c0c0777034c/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:242 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09d024f0-dc43-4c99-81dd-ec9a2c126e86/volumes/kubernetes.io~projected/kube-api-access-bsk5z DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/aff21346-754f-4de1-b935-26f9a51145fe/volumes/kubernetes.io~projected/kube-api-access-n8xqx DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4b2ec9adbb917d9205f405a968e2677147436e0f71b66e5e17b1a27a7c548b4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/991476c7-c67c-436a-83e0-e1514350d019/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-188 DeviceMajor:0 DeviceMinor:188 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0bd6859-5176-430c-88cc-507c3f6c136b/volumes/kubernetes.io~projected/kube-api-access-vln5d DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/533f521b860f11ff4902df67f658c3bd6cdf11671160b32dee2732a566477840/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/394dc6a3f2d75ae5fa75894c611947fb11bff76f9c1d8623cf765704782349ab/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64e43900-4506-4cca-88fc-8da6c38f07ac/volumes/kubernetes.io~projected/kube-api-access-q548s DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c697a3308b3ce54e0c1b9bfc7076c6884fedbc1167b148c7634b0a59e462bb7/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e5ac8782-ecd9-4940-85a8-af444cbb0a74/volumes/kubernetes.io~projected/kube-api-access-2rlr5 DeviceMajor:0 DeviceMinor:233 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9b32d7-fb57-41c2-8b64-e5e4a2449169/volumes/kubernetes.io~projected/kube-api-access-kwqrl DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb90ff60287a8c76afe91c2493cd641894c8a83163328042c822928d2f886ff4/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b49ef0f6-2ac3-4393-b45a-5356110e13a3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9b32d7-fb57-41c2-8b64-e5e4a2449169/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd77f956d4d408cdef442b8359f3a857b52b91945cbbd8648a6696c3a944d73b/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4f55d6-e263-446d-bb7a-884cc4f1c74c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4f55d6-e263-446d-bb7a-884cc4f1c74c/volumes/kubernetes.io~projected/kube-api-access-czn8g DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9c53ce77-00c9-4509-9650-115117f93d54/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/991476c7-c67c-436a-83e0-e1514350d019/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf9beffb-08c3-4766-bdd8-cdeaa557e5be/volumes/kubernetes.io~projected/kube-api-access-z4w7s DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc38350aca267c87ffdd73d6f139252979c0d21a3017070c4ed216d04ba790db/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcd162a8916ef90eb06ed9773f93b08a716748efea7d71f9a8ce9317ecf33f35/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da2aa0ce7d11307030dcc28b4bcfdbf08454aabdc358d41f0aa2bdf913d8df92/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94a990d84d550b20559475c37c1036a4f63c2cf784d73674f4b782ec8b1e14a8/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/83f38264-bee5-4874-8bb0-e8ab38d21598/volumes/kubernetes.io~projected/kube-api-access-b2q57 DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d8115cff4151270a2a340c4c815e4edfa493fd575ee25208b986bad1f29f160/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12cd2b00-faff-422b-a588-c07050ab96f7/volumes/kubernetes.io~projected/kube-api-access-rflhg DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/991476c7-c67c-436a-83e0-e1514350d019/volumes/kubernetes.io~projected/kube-api-access-jm6g9 DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2/volumes/kubernetes.io~projected/kube-api-access-scdz6 DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/193ad4d9-4ef2-4ede-abd6-6ab5b589321a/volumes/kubernetes.io~projected/kube-api-access-ppstg DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6dd64c8ab1801f80a647e9603e93cf037daa5e56a3626056c0b2e9a8fdffc03/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/e3d1ef12-c282-4cba-9678-e369daacc3d1/volumes/kubernetes.io~projected/kube-api-access-mjrdf DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb7e36d875c48455585d287c440fe02e2ec49a2726b9c9de68581ba051e4445c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b9b32d7-fb57-41c2-8b64-e5e4a2449169/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/83f38264-bee5-4874-8bb0-e8ab38d21598/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2c697a3308b3ce5 MacAddress:4a:fb:3b:52:a2:36 Speed:10000 Mtu:8901} {Name:394dc6a3f2d75ae MacAddress:4a:74:4f:60:e2:2e Speed:10000 Mtu:8901} {Name:94a990d84d550b2 MacAddress:06:32:c5:77:65:9d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cd:67:d4:4b Speed:0 Mtu:9001} {Name:br-int MacAddress:de:9e:ab:36:e4:b0 Speed:0 Mtu:8901} {Name:d3ca4c9fa5be461 MacAddress:ee:11:8a:20:b5:f5 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cd:67:d4:4b Speed:0 Mtu:9001} {Name:f0c8b4a7fa3357d MacAddress:22:91:0c:9f:d6:b4 Speed:10000 Mtu:8901} {Name:fcd162a8916ef90 MacAddress:fa:2f:60:fb:30:c3 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:0e:bf:5a:e9:91:02 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:22:d6:45:23:38:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.930600    4550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.930642    4550 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.931166    4550 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.931192    4550 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-85.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.931512    4550 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.931525    4550 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.931540    4550 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.931558    4550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.931656    4550 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.931889    4550 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.932177    4550 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.932415    4550 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.932437    4550 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.932452    4550 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.932469    4550 kubelet.go:405] "Adding apiserver pod source"
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.932481    4550 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.933258    4550 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.933282    4550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 21:46:23.936099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.933795    4550 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 21:46:23.936734 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936496    4550 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 21:46:23.936734 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936506    4550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 21:46:23.936734 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936611    4550 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 29 21:46:23.936816 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936782    4550 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 21:46:23.936816 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936809    4550 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 21:46:23.936901 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936890    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 21:46:23.936945 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936906    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 21:46:23.936945 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936912    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 21:46:23.936945 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936917    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 21:46:23.936945 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936923    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 21:46:23.936945 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936928    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 21:46:23.936945 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936933    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 21:46:23.936945 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936942    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 21:46:23.937185 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936951    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 21:46:23.937185 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936962    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 21:46:23.937185 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936970    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 21:46:23.937185 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.936980    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 21:46:23.937185 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.937007    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 21:46:23.937185 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.937016    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 21:46:23.940250 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.940235    4550 server.go:1263] "Started kubelet"
Oct 29 21:46:23.940483 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.940408    4550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 21:46:23.940571 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.940493    4550 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 21:46:23.940638 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.940445    4550 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 21:46:23.941993 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.940783    4550 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 21:46:23.940775 ip-10-0-128-85 systemd[1]: Started Kubernetes Kubelet.
Oct 29 21:46:23.942221 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.942202    4550 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 21:46:23.943873 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.943840    4550 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 21:46:23.946780 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:23.946268    4550 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-85.ec2.internal\" not found"
Oct 29 21:46:23.946780 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.946320    4550 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 21:46:23.946780 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.946328    4550 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 21:46:23.946780 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.946434    4550 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 21:46:23.946780 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.946709    4550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 21:46:23.946780 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.946757    4550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:40:46 +0000 UTC" deadline="2028-06-20 08:46:54.965527528 +0000 UTC"
Oct 29 21:46:23.947130 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.946787    4550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14387h0m31.018743815s"
Oct 29 21:46:23.947851 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.947743    4550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 21:46:23.947851 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.947760    4550 factory.go:55] Registering systemd factory
Oct 29 21:46:23.947851 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.947780    4550 factory.go:223] Registration of the systemd container factory successfully
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948156    4550 factory.go:153] Registering CRI-O factory
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948182    4550 factory.go:223] Registration of the crio container factory successfully
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948211    4550 factory.go:103] Registering Raw factory
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948230    4550 manager.go:1199] Started watching for new ooms in manager
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948671    4550 manager.go:319] Starting recovery of all containers
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948740    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aff21346-754f-4de1-b935-26f9a51145fe" volumeName="kubernetes.io/projected/aff21346-754f-4de1-b935-26f9a51145fe-kube-api-access-n8xqx" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948798    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3d1ef12-c282-4cba-9678-e369daacc3d1" volumeName="kubernetes.io/configmap/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-daemon-config" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948808    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="193ad4d9-4ef2-4ede-abd6-6ab5b589321a" volumeName="kubernetes.io/configmap/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-iptables-alerter-script" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948826    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9b32d7-fb57-41c2-8b64-e5e4a2449169" volumeName="kubernetes.io/projected/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-kube-api-access-kwqrl" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948835    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c53ce77-00c9-4509-9650-115117f93d54" volumeName="kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948844    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2" volumeName="kubernetes.io/projected/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-kube-api-access-scdz6" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948861    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce4f55d6-e263-446d-bb7a-884cc4f1c74c" volumeName="kubernetes.io/projected/ce4f55d6-e263-446d-bb7a-884cc4f1c74c-kube-api-access-czn8g" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948871    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3d1ef12-c282-4cba-9678-e369daacc3d1" volumeName="kubernetes.io/configmap/e3d1ef12-c282-4cba-9678-e369daacc3d1-cni-binary-copy" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948890    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12cd2b00-faff-422b-a588-c07050ab96f7" volumeName="kubernetes.io/projected/12cd2b00-faff-422b-a588-c07050ab96f7-kube-api-access-rflhg" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948914    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84b63148-f00f-4506-b07d-9c0c0777034c" volumeName="kubernetes.io/empty-dir/84b63148-f00f-4506-b07d-9c0c0777034c-tmp-dir" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948933    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="991476c7-c67c-436a-83e0-e1514350d019" volumeName="kubernetes.io/projected/991476c7-c67c-436a-83e0-e1514350d019-kube-api-access-jm6g9" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948942    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0bd6859-5176-430c-88cc-507c3f6c136b" volumeName="kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948986    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b49ef0f6-2ac3-4393-b45a-5356110e13a3" volumeName="kubernetes.io/secret/b49ef0f6-2ac3-4393-b45a-5356110e13a3-agent-certs" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.948998    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09d024f0-dc43-4c99-81dd-ec9a2c126e86" volumeName="kubernetes.io/empty-dir/09d024f0-dc43-4c99-81dd-ec9a2c126e86-tmp-dir" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.949021    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9b32d7-fb57-41c2-8b64-e5e4a2449169" volumeName="kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovnkube-script-lib" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.949051    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83f38264-bee5-4874-8bb0-e8ab38d21598" volumeName="kubernetes.io/projected/83f38264-bee5-4874-8bb0-e8ab38d21598-kube-api-access-b2q57" seLinuxMountContext=""
Oct 29 21:46:23.949158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.949072    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5ac8782-ecd9-4940-85a8-af444cbb0a74" volumeName="kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953124    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0bd6859-5176-430c-88cc-507c3f6c136b" volumeName="kubernetes.io/empty-dir/a0bd6859-5176-430c-88cc-507c3f6c136b-data-volume" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953179    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" volumeName="kubernetes.io/projected/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-kube-api-access-z4w7s" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953196    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84b63148-f00f-4506-b07d-9c0c0777034c" volumeName="kubernetes.io/configmap/84b63148-f00f-4506-b07d-9c0c0777034c-config-volume" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953208    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84b63148-f00f-4506-b07d-9c0c0777034c" volumeName="kubernetes.io/projected/84b63148-f00f-4506-b07d-9c0c0777034c-kube-api-access-pbdjz" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953220    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84b63148-f00f-4506-b07d-9c0c0777034c" volumeName="kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953233    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="991476c7-c67c-436a-83e0-e1514350d019" volumeName="kubernetes.io/empty-dir/991476c7-c67c-436a-83e0-e1514350d019-tmp" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953245    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" volumeName="kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953257    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3d1ef12-c282-4cba-9678-e369daacc3d1" volumeName="kubernetes.io/projected/e3d1ef12-c282-4cba-9678-e369daacc3d1-kube-api-access-mjrdf" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953268    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09d024f0-dc43-4c99-81dd-ec9a2c126e86" volumeName="kubernetes.io/projected/09d024f0-dc43-4c99-81dd-ec9a2c126e86-kube-api-access-bsk5z" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953280    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64e43900-4506-4cca-88fc-8da6c38f07ac" volumeName="kubernetes.io/projected/64e43900-4506-4cca-88fc-8da6c38f07ac-kube-api-access-q548s" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953292    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83f38264-bee5-4874-8bb0-e8ab38d21598" volumeName="kubernetes.io/secret/83f38264-bee5-4874-8bb0-e8ab38d21598-signing-key" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953329    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b49ef0f6-2ac3-4393-b45a-5356110e13a3" volumeName="kubernetes.io/configmap/b49ef0f6-2ac3-4393-b45a-5356110e13a3-konnectivity-ca" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953340    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" volumeName="kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cni-binary-copy" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953352    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="193ad4d9-4ef2-4ede-abd6-6ab5b589321a" volumeName="kubernetes.io/projected/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-kube-api-access-ppstg" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953365    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9b32d7-fb57-41c2-8b64-e5e4a2449169" volumeName="kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-env-overrides" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953377    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83f38264-bee5-4874-8bb0-e8ab38d21598" volumeName="kubernetes.io/configmap/83f38264-bee5-4874-8bb0-e8ab38d21598-signing-cabundle" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953389    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0bd6859-5176-430c-88cc-507c3f6c136b" volumeName="kubernetes.io/projected/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-api-access-vln5d" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953401    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" volumeName="kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953429    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce4f55d6-e263-446d-bb7a-884cc4f1c74c" volumeName="kubernetes.io/secret/ce4f55d6-e263-446d-bb7a-884cc4f1c74c-cert" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953441    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9b32d7-fb57-41c2-8b64-e5e4a2449169" volumeName="kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovnkube-config" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953456    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b9b32d7-fb57-41c2-8b64-e5e4a2449169" volumeName="kubernetes.io/secret/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953469    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="991476c7-c67c-436a-83e0-e1514350d019" volumeName="kubernetes.io/empty-dir/991476c7-c67c-436a-83e0-e1514350d019-etc-tuned" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953481    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2" volumeName="kubernetes.io/configmap/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-serviceca" seLinuxMountContext=""
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953496    4550 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 21:46:23.954404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.953504    4550 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 21:46:23.964370 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.964067    4550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 21:46:23.980741 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.980716    4550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 21:46:23.980741 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.980738    4550 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 21:46:23.980903 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:23.980757    4550 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 21:46:23.980903 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:23.980799    4550 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 21:46:24.008010 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.007981    4550 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:46:24.008190 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.008084    4550 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 29 21:46:24.008269 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.008215    4550 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:46:24.008353 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.008336    4550 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:46:24.021403 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.021378    4550 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:46:24.029476 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.029427    4550 generic.go:356] "Generic (PLEG): container finished" podID="cc03cae4f4670aad3e1f4f4e61ad2796" containerID="133ba2f2253133b948cf2f041a4adea29468a6eb5d958ae281d91cfc8338e78d" exitCode=0
Oct 29 21:46:24.045712 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.045689    4550 manager.go:324] Recovery completed
Oct 29 21:46:24.048244 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.048217    4550 generic.go:356] "Generic (PLEG): container finished" podID="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" containerID="3199d7abd32e1f5320633223311c9298783ace57d11588b2dcfc0d3eb12cab43" exitCode=0
Oct 29 21:46:24.048244 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.048242    4550 generic.go:356] "Generic (PLEG): container finished" podID="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" containerID="e93981ed89981a2e3b96ed0c0e6e85fca8a4c900e53985ec48a4506985ba6ab9" exitCode=0
Oct 29 21:46:24.048354 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.048250    4550 generic.go:356] "Generic (PLEG): container finished" podID="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" containerID="c345f0962c58bcdc400cfd888c9f2ec90882afb7318f8928b312a352ad90bfaf" exitCode=0
Oct 29 21:46:24.048354 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.048257    4550 generic.go:356] "Generic (PLEG): container finished" podID="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" containerID="750e8ac814df33179779b6d8bfc6417ff385a6d479d61696bb45ad6c69a548e3" exitCode=0
Oct 29 21:46:24.048354 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.048262    4550 generic.go:356] "Generic (PLEG): container finished" podID="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" containerID="0ab5015a1edf8b24fa2f2a0c931328d31abaff156dbc33d29f3bad611643dcfa" exitCode=0
Oct 29 21:46:24.048354 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.048268    4550 generic.go:356] "Generic (PLEG): container finished" podID="bf9beffb-08c3-4766-bdd8-cdeaa557e5be" containerID="cf6f712c999e6f5148e00c87356289885b68ab5f793704c9debd115cedf3fdce" exitCode=0
Oct 29 21:46:24.065498 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.065444    4550 cpu_manager.go:226] "Starting" policy="none"
Oct 29 21:46:24.065498 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.065458    4550 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 21:46:24.065498 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.065478    4550 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 21:46:24.065686 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.065657    4550 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 29 21:46:24.065739 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.065675    4550 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 29 21:46:24.065739 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.065707    4550 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 29 21:46:24.065739 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.065716    4550 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 29 21:46:24.065739 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.065724    4550 policy_none.go:50] "Start"
Oct 29 21:46:24.065739 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.065738    4550 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 21:46:24.065967 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.065752    4550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 21:46:24.065967 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.065938    4550 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 29 21:46:24.065967 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.065949    4550 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 29 21:46:24.065967 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.065957    4550 policy_none.go:44] "Start"
Oct 29 21:46:24.069953 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.069923    4550 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 21:46:24.073539 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.073520    4550 manager.go:341] "Starting Device Plugin manager"
Oct 29 21:46:24.075005 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:24.074019    4550 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 21:46:24.075005 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.074038    4550 server.go:87] "Starting device plugin registration server"
Oct 29 21:46:24.075005 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.074353    4550 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 21:46:24.075005 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.074366    4550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 21:46:24.075005 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.074741    4550 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 21:46:24.075005 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.074819    4550 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 21:46:24.075005 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.074850    4550 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 21:46:24.075005 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.074855    4550 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 21:46:24.075005 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.074931    4550 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T21:46:24.074841563Z","UUID":"32c84311-34f4-4af8-80b8-32fe644d64d1","Handler":null,"Name":"","Endpoint":""}
Oct 29 21:46:24.076354 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.076334    4550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 21:46:24.076448 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.076361    4550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 21:46:24.076577 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:24.076561    4550 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 21:46:24.081104 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.081036    4550 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal"]
Oct 29 21:46:24.082633 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.082617    4550 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.082747 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.082615    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal" event={"ID":"d33928eb6a1a12294cb01f2c571c879b","Type":"ContainerStarted","Data":"2460c3979229597df539441cc69fa3d1235d3b36646e237203a3b687e99827e3"}
Oct 29 21:46:24.082747 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.082658    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal" event={"ID":"d33928eb6a1a12294cb01f2c571c879b","Type":"ContainerStarted","Data":"0d8115cff4151270a2a340c4c815e4edfa493fd575ee25208b986bad1f29f160"}
Oct 29 21:46:24.082747 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.082688    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal" event={"ID":"cc03cae4f4670aad3e1f4f4e61ad2796","Type":"ContainerStarted","Data":"627feaaa7af5d031b654f3cfb53f6e1be61041430e9006d7721de3833a647897"}
Oct 29 21:46:24.082747 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.082706    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal" event={"ID":"cc03cae4f4670aad3e1f4f4e61ad2796","Type":"ContainerDied","Data":"133ba2f2253133b948cf2f041a4adea29468a6eb5d958ae281d91cfc8338e78d"}
Oct 29 21:46:24.082747 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.082724    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal" event={"ID":"cc03cae4f4670aad3e1f4f4e61ad2796","Type":"ContainerStarted","Data":"c4b2ec9adbb917d9205f405a968e2677147436e0f71b66e5e17b1a27a7c548b4"}
Oct 29 21:46:24.082983 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.082860    4550 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.107072 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.107052    4550 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:46:24.107171 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:24.107089    4550 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-85.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.107871 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.107859    4550 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:46:24.107941 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:24.107884    4550 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.175457 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.175433    4550 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:46:24.176743 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.176727    4550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:46:24.176825 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.176754    4550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:46:24.176825 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.176771    4550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-85.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:46:24.176825 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.176806    4550 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.209673 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.209650    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc03cae4f4670aad3e1f4f4e61ad2796-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal\" (UID: \"cc03cae4f4670aad3e1f4f4e61ad2796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.209735 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.209674    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc03cae4f4670aad3e1f4f4e61ad2796-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal\" (UID: \"cc03cae4f4670aad3e1f4f4e61ad2796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.209735 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.209691    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d33928eb6a1a12294cb01f2c571c879b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-85.ec2.internal\" (UID: \"d33928eb6a1a12294cb01f2c571c879b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.220284 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.220264    4550 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.220354 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.220327    4550 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.310135 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.310109    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc03cae4f4670aad3e1f4f4e61ad2796-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal\" (UID: \"cc03cae4f4670aad3e1f4f4e61ad2796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.310135 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.310135    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc03cae4f4670aad3e1f4f4e61ad2796-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal\" (UID: \"cc03cae4f4670aad3e1f4f4e61ad2796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.310293 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.310165    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d33928eb6a1a12294cb01f2c571c879b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-85.ec2.internal\" (UID: \"d33928eb6a1a12294cb01f2c571c879b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.310293 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.310220    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc03cae4f4670aad3e1f4f4e61ad2796-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal\" (UID: \"cc03cae4f4670aad3e1f4f4e61ad2796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.310293 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.310220    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc03cae4f4670aad3e1f4f4e61ad2796-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal\" (UID: \"cc03cae4f4670aad3e1f4f4e61ad2796\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.310293 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.310274    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d33928eb6a1a12294cb01f2c571c879b-config\") pod \"kube-apiserver-proxy-ip-10-0-128-85.ec2.internal\" (UID: \"d33928eb6a1a12294cb01f2c571c879b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:24.933313 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.933284    4550 apiserver.go:52] "Watching apiserver"
Oct 29 21:46:24.974034 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.974008    4550 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 21:46:24.974454 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.974433    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-rx95w","openshift-service-ca/service-ca-5fffc97bdf-h7nlm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs","kube-system/global-pull-secret-syncer-7rvg5","openshift-multus/network-metrics-daemon-t68qw","openshift-network-diagnostics/network-check-target-hvfh7","openshift-ingress-canary/ingress-canary-sn7rn","kube-system/konnectivity-agent-4hxw6","kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-j8x4s","openshift-insights/insights-runtime-extractor-d8bd7","openshift-ovn-kubernetes/ovnkube-node-mx2lj","openshift-dns/node-resolver-6bf42","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal","openshift-network-operator/iptables-alerter-crv8b","openshift-cluster-node-tuning-operator/tuned-zqvj9","openshift-multus/multus-additional-cni-plugins-4kdn5","openshift-multus/multus-zmrms","openshift-dns/dns-default-x5mwg"]
Oct 29 21:46:24.975947 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.975921    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:24.983180 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.983157    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:24.989549 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.989531    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 21:46:24.989982 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.989967    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ckcdt\""
Oct 29 21:46:24.992658 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.992642    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 21:46:24.992704 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.992653    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 21:46:24.992704 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.992673    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 21:46:24.992771 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:24.992725    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-znmmx\""
Oct 29 21:46:25.001286 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.001268    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 21:46:25.001286 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.001284    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 21:46:25.001390 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.001348    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5fns5\""
Oct 29 21:46:25.001784 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.001770    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 21:46:25.002063 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.002052    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 21:46:25.002122 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.002060    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:25.002192 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.002121    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 21:46:25.002366 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.002353    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 21:46:25.002450 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.002434    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 21:46:25.002499 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.002438    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 21:46:25.002499 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.002455    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 21:46:25.002559 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.002496    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:25.002634 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.002618    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 21:46:25.002664 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.002625    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 21:46:25.002914 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.002898    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hzxgj\""
Oct 29 21:46:25.002960 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.002921    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l6fj9\""
Oct 29 21:46:25.002960 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.002933    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7dqcg\""
Oct 29 21:46:25.003027 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.002968    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6chnm\""
Oct 29 21:46:25.003458 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.003443    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4sldf\""
Oct 29 21:46:25.003701 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.003685    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 29 21:46:25.003854 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.003692    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 21:46:25.003941 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.003924    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 21:46:25.003978 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.003939    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8ld6j\""
Oct 29 21:46:25.004644 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.004627    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 21:46:25.004703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.004641    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 21:46:25.004703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.004659    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w9rdw\""
Oct 29 21:46:25.004703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.004687    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 29 21:46:25.004908 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.004886    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hxxfk\""
Oct 29 21:46:25.004982 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.004954    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pvv78\""
Oct 29 21:46:25.005027 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.005014    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 21:46:25.005330 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.005313    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 21:46:25.005426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.005329    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 21:46:25.005426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.005315    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 21:46:25.005426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.005348    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 21:46:25.005426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.005334    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 21:46:25.005426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.005346    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6jh7q\""
Oct 29 21:46:25.005426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.005375    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 21:46:25.005426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.005375    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 21:46:25.005426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.005375    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 29 21:46:25.005426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.005396    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 21:46:25.005426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.005398    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-p5fxq\""
Oct 29 21:46:25.005426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.005392    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 21:46:25.005426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.005426    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:25.005758 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.005473    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mlr8g\""
Oct 29 21:46:25.013745 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.013726    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 29 21:46:25.014014 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.013988    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 21:46:25.014014 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.013994    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsffp\""
Oct 29 21:46:25.014166 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.014007    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 29 21:46:25.014166 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.014030    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 21:46:25.014166 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.014065    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 21:46:25.014166 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.014070    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 21:46:25.014336 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.014286    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 21:46:25.014336 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.014295    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 21:46:25.014336 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.014286    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 21:46:25.014336 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.014318    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 21:46:25.014336 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.014323    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vdqjz\""
Oct 29 21:46:25.014496 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.014418    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 21:46:25.047784 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.047756    4550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 21:46:25.053103 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.053080    4550 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:25.053188 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.053159    4550 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:25.076838 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.076819    4550 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:46:25.076902 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:25.076855    4550 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-85.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:25.077485 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.077473    4550 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:46:25.077521 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:25.077502    4550 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-85.ec2.internal"
Oct 29 21:46:25.114120 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114095    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-host\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:46:25.114217 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114124    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-cni-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.114217 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114149    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-cnibin\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.114217 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114165    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-lib-modules\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.114316 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114217    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-host\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.114316 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114255    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-etc-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.114316 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114284    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q548s\" (UniqueName: \"kubernetes.io/projected/64e43900-4506-4cca-88fc-8da6c38f07ac-kube-api-access-q548s\") pod \"migrator-66df5d496b-j8x4s\" (UID: \"64e43900-4506-4cca-88fc-8da6c38f07ac\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-j8x4s"
Oct 29 21:46:25.114421 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114316    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.114459 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114440    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-netns\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.114493 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114468    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-cni-bin\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.114493 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114486    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-systemd\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.114583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114502    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.114583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114537    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-device-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.114674 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114596    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.114674 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114642    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cni-binary-copy\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.114674 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114645    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4w7s\" (UniqueName: \"kubernetes.io/projected/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-kube-api-access-z4w7s\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.114822 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114682    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0bd6859-5176-430c-88cc-507c3f6c136b-data-volume\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:25.114822 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114721    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-systemd-units\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.114822 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114748    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovnkube-script-lib\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.114822 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114776    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsk5z\" (UniqueName: \"kubernetes.io/projected/09d024f0-dc43-4c99-81dd-ec9a2c126e86-kube-api-access-bsk5z\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:46:25.114822 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114803    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.114822 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114812    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0bd6859-5176-430c-88cc-507c3f6c136b-data-volume\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:25.115110 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114855    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.115110 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114876    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-var-lib-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.115110 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114891    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-hostroot\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.115110 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114936    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-conf-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.115110 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.114973    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppstg\" (UniqueName: \"kubernetes.io/projected/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-kube-api-access-ppstg\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:46:25.115110 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115096    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-modprobe-d\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.115365 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115115    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.115365 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115176    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm6g9\" (UniqueName: \"kubernetes.io/projected/991476c7-c67c-436a-83e0-e1514350d019-kube-api-access-jm6g9\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.115365 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115182    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovnkube-script-lib\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.115365 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115207    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.115365 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115310    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-log-socket\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.115365 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115348    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-socket-dir-parent\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.115621 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115410    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-cni-multus\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.115621 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115449    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-kubernetes\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.115621 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115486    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-sys-fs\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.115621 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115510    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflhg\" (UniqueName: \"kubernetes.io/projected/12cd2b00-faff-422b-a588-c07050ab96f7-kube-api-access-rflhg\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.115621 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115588    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:25.115829 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115628    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-iptables-alerter-script\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:46:25.115829 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115712    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84b63148-f00f-4506-b07d-9c0c0777034c-tmp-dir\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:25.115829 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115742    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/991476c7-c67c-436a-83e0-e1514350d019-etc-tuned\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.115829 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115769    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-registration-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.115829 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115795    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/83f38264-bee5-4874-8bb0-e8ab38d21598-signing-key\") pod \"service-ca-5fffc97bdf-h7nlm\" (UID: \"83f38264-bee5-4874-8bb0-e8ab38d21598\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:25.115829 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115814    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:25.115829 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115825    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84b63148-f00f-4506-b07d-9c0c0777034c-tmp-dir\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:25.116036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115852    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-node-log\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.116036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115868    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.116036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115888    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scdz6\" (UniqueName: \"kubernetes.io/projected/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-kube-api-access-scdz6\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:46:25.116036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115903    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/991476c7-c67c-436a-83e0-e1514350d019-tmp\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.116036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115907    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-iptables-alerter-script\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:46:25.116036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115916    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-kubelet\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.116036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115939    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-etc-kubernetes\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.116036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115966    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:25.116036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.115996    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlr5\" (UniqueName: \"kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5\") pod \"network-check-target-hvfh7\" (UID: \"e5ac8782-ecd9-4940-85a8-af444cbb0a74\") " pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:25.116036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116018    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/991476c7-c67c-436a-83e0-e1514350d019-etc-tuned\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.116036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116021    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/991476c7-c67c-436a-83e0-e1514350d019-tmp\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.116487 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116061    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysconfig\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.116487 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116081    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/83f38264-bee5-4874-8bb0-e8ab38d21598-signing-key\") pod \"service-ca-5fffc97bdf-h7nlm\" (UID: \"83f38264-bee5-4874-8bb0-e8ab38d21598\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:25.116487 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116092    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czn8g\" (UniqueName: \"kubernetes.io/projected/ce4f55d6-e263-446d-bb7a-884cc4f1c74c-kube-api-access-czn8g\") pod \"ingress-canary-sn7rn\" (UID: \"ce4f55d6-e263-446d-bb7a-884cc4f1c74c\") " pod="openshift-ingress-canary/ingress-canary-sn7rn"
Oct 29 21:46:25.116487 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116126    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cnibin\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.116487 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116234    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vln5d\" (UniqueName: \"kubernetes.io/projected/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-api-access-vln5d\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:25.116487 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116280    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-host-slash\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:46:25.116487 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116376    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c53ce77-00c9-4509-9650-115117f93d54-original-pull-secret\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:25.116487 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116384    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:25.116487 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116445    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c53ce77-00c9-4509-9650-115117f93d54-kubelet-config\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:25.116487 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116476    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysctl-d\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.116897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116504    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-sys\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.116897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116529    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-systemd\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.116897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116553    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovnkube-config\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.116897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116578    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-serviceca\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:46:25.116897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116674    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3d1ef12-c282-4cba-9678-e369daacc3d1-cni-binary-copy\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.116897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116723    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c53ce77-00c9-4509-9650-115117f93d54-dbus\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:25.116897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116738    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovnkube-config\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.116897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116753    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-serviceca\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:46:25.116897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116767    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b49ef0f6-2ac3-4393-b45a-5356110e13a3-agent-certs\") pod \"konnectivity-agent-4hxw6\" (UID: \"b49ef0f6-2ac3-4393-b45a-5356110e13a3\") " pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:46:25.116897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116810    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2q57\" (UniqueName: \"kubernetes.io/projected/83f38264-bee5-4874-8bb0-e8ab38d21598-kube-api-access-b2q57\") pod \"service-ca-5fffc97bdf-h7nlm\" (UID: \"83f38264-bee5-4874-8bb0-e8ab38d21598\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:25.116897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116826    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-run-netns\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.116897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116840    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.116897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116874    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovn-node-metrics-cert\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.117398 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116912    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwqrl\" (UniqueName: \"kubernetes.io/projected/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-kube-api-access-kwqrl\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.117398 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116913    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3d1ef12-c282-4cba-9678-e369daacc3d1-cni-binary-copy\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.117398 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.116949    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b49ef0f6-2ac3-4393-b45a-5356110e13a3-agent-certs\") pod \"konnectivity-agent-4hxw6\" (UID: \"b49ef0f6-2ac3-4393-b45a-5356110e13a3\") " pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:46:25.117398 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.117019    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-ovn-node-metrics-cert\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.117398 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.117030    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrdf\" (UniqueName: \"kubernetes.io/projected/e3d1ef12-c282-4cba-9678-e369daacc3d1-kube-api-access-mjrdf\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.117398 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.117057    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:25.117398 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.117089    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbdjz\" (UniqueName: \"kubernetes.io/projected/84b63148-f00f-4506-b07d-9c0c0777034c-kube-api-access-pbdjz\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:25.117398 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.117111    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-system-cni-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.117398 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.117126    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-k8s-cni-cncf-io\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.117783 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.117618    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d024f0-dc43-4c99-81dd-ec9a2c126e86-tmp-dir\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:46:25.117783 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.117648    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d024f0-dc43-4c99-81dd-ec9a2c126e86-tmp-dir\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:46:25.117783 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.117667    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-run\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.117783 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.117652    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84b63148-f00f-4506-b07d-9c0c0777034c-metrics-tls\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:25.117783 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.117695    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-var-lib-kubelet\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.117783 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.117753    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce4f55d6-e263-446d-bb7a-884cc4f1c74c-cert\") pod \"ingress-canary-sn7rn\" (UID: \"ce4f55d6-e263-446d-bb7a-884cc4f1c74c\") " pod="openshift-ingress-canary/ingress-canary-sn7rn"
Oct 29 21:46:25.118048 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.117793    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-socket-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.118048 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.117826    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-daemon-config\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.118048 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.117866    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8xqx\" (UniqueName: \"kubernetes.io/projected/aff21346-754f-4de1-b935-26f9a51145fe-kube-api-access-n8xqx\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:25.118224 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118199    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b49ef0f6-2ac3-4393-b45a-5356110e13a3-konnectivity-ca\") pod \"konnectivity-agent-4hxw6\" (UID: \"b49ef0f6-2ac3-4393-b45a-5356110e13a3\") " pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:46:25.118341 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118322    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b49ef0f6-2ac3-4393-b45a-5356110e13a3-konnectivity-ca\") pod \"konnectivity-agent-4hxw6\" (UID: \"b49ef0f6-2ac3-4393-b45a-5356110e13a3\") " pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:46:25.118341 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118335    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce4f55d6-e263-446d-bb7a-884cc4f1c74c-cert\") pod \"ingress-canary-sn7rn\" (UID: \"ce4f55d6-e263-446d-bb7a-884cc4f1c74c\") " pod="openshift-ingress-canary/ingress-canary-sn7rn"
Oct 29 21:46:25.118474 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118372    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysctl-conf\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.118474 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118406    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-daemon-config\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.118474 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118416    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-system-cni-dir\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.118613 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118499    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.118613 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118539    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0bd6859-5176-430c-88cc-507c3f6c136b-crio-socket\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:25.118613 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118571    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:25.118753 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118614    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/83f38264-bee5-4874-8bb0-e8ab38d21598-signing-cabundle\") pod \"service-ca-5fffc97bdf-h7nlm\" (UID: \"83f38264-bee5-4874-8bb0-e8ab38d21598\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:25.118753 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118643    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-kubelet\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.118753 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118709    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-ovn\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.118753 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118739    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-cni-bin\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.118753 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118743    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.118965 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118768    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-cni-netd\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.118965 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118836    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-multus-certs\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.118965 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.118943    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84b63148-f00f-4506-b07d-9c0c0777034c-config-volume\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:25.119086 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.119003    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09d024f0-dc43-4c99-81dd-ec9a2c126e86-hosts-file\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:46:25.119086 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.119013    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/83f38264-bee5-4874-8bb0-e8ab38d21598-signing-cabundle\") pod \"service-ca-5fffc97bdf-h7nlm\" (UID: \"83f38264-bee5-4874-8bb0-e8ab38d21598\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:25.119086 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.119060    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-slash\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.119238 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.119129    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-env-overrides\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.119281 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.119264    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84b63148-f00f-4506-b07d-9c0c0777034c-config-volume\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:25.119421 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.119393    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-env-overrides\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.136122 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.136099    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q548s\" (UniqueName: \"kubernetes.io/projected/64e43900-4506-4cca-88fc-8da6c38f07ac-kube-api-access-q548s\") pod \"migrator-66df5d496b-j8x4s\" (UID: \"64e43900-4506-4cca-88fc-8da6c38f07ac\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-j8x4s"
Oct 29 21:46:25.136286 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.136267    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsk5z\" (UniqueName: \"kubernetes.io/projected/09d024f0-dc43-4c99-81dd-ec9a2c126e86-kube-api-access-bsk5z\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:46:25.136621 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.136604    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppstg\" (UniqueName: \"kubernetes.io/projected/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-kube-api-access-ppstg\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:46:25.136663 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.136622    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm6g9\" (UniqueName: \"kubernetes.io/projected/991476c7-c67c-436a-83e0-e1514350d019-kube-api-access-jm6g9\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.137538 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.137509    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflhg\" (UniqueName: \"kubernetes.io/projected/12cd2b00-faff-422b-a588-c07050ab96f7-kube-api-access-rflhg\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.137721 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.137705    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scdz6\" (UniqueName: \"kubernetes.io/projected/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-kube-api-access-scdz6\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:46:25.137762 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.137743    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4w7s\" (UniqueName: \"kubernetes.io/projected/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-kube-api-access-z4w7s\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.137903 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.137886    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czn8g\" (UniqueName: \"kubernetes.io/projected/ce4f55d6-e263-446d-bb7a-884cc4f1c74c-kube-api-access-czn8g\") pod \"ingress-canary-sn7rn\" (UID: \"ce4f55d6-e263-446d-bb7a-884cc4f1c74c\") " pod="openshift-ingress-canary/ingress-canary-sn7rn"
Oct 29 21:46:25.138544 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.138527    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vln5d\" (UniqueName: \"kubernetes.io/projected/a0bd6859-5176-430c-88cc-507c3f6c136b-kube-api-access-vln5d\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:25.138605 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.138591    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2q57\" (UniqueName: \"kubernetes.io/projected/83f38264-bee5-4874-8bb0-e8ab38d21598-kube-api-access-b2q57\") pod \"service-ca-5fffc97bdf-h7nlm\" (UID: \"83f38264-bee5-4874-8bb0-e8ab38d21598\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h7nlm"
Oct 29 21:46:25.139120 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.139105    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrdf\" (UniqueName: \"kubernetes.io/projected/e3d1ef12-c282-4cba-9678-e369daacc3d1-kube-api-access-mjrdf\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.139210 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.139190    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8xqx\" (UniqueName: \"kubernetes.io/projected/aff21346-754f-4de1-b935-26f9a51145fe-kube-api-access-n8xqx\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:25.139345 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.139325    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rlr5\" (UniqueName: \"kubernetes.io/projected/e5ac8782-ecd9-4940-85a8-af444cbb0a74-kube-api-access-2rlr5\") pod \"network-check-target-hvfh7\" (UID: \"e5ac8782-ecd9-4940-85a8-af444cbb0a74\") " pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:25.141019 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.141000    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbdjz\" (UniqueName: \"kubernetes.io/projected/84b63148-f00f-4506-b07d-9c0c0777034c-kube-api-access-pbdjz\") pod \"dns-default-x5mwg\" (UID: \"84b63148-f00f-4506-b07d-9c0c0777034c\") " pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:25.141382 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.141367    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwqrl\" (UniqueName: \"kubernetes.io/projected/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-kube-api-access-kwqrl\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.219361 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219309    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-systemd\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.219361 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219335    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c53ce77-00c9-4509-9650-115117f93d54-dbus\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:25.219361 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219351    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-run-netns\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.219501 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219360    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-systemd\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.219501 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219390    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.219501 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219364    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.219501 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219435    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c53ce77-00c9-4509-9650-115117f93d54-dbus\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:25.219501 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219436    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-system-cni-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.219501 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219459    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-run-netns\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.219501 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219464    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-k8s-cni-cncf-io\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.219501 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219490    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-system-cni-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.219501 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219489    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-run\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219513    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-var-lib-kubelet\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219528    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-socket-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219533    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-run\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219544    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysctl-conf\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219563    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-system-cni-dir\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219576    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-k8s-cni-cncf-io\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219579    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0bd6859-5176-430c-88cc-507c3f6c136b-crio-socket\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219601    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0bd6859-5176-430c-88cc-507c3f6c136b-crio-socket\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219616    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219628    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-var-lib-kubelet\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219644    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-kubelet\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219670    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-ovn\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219693    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-cni-bin\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219700    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysctl-conf\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219718    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-cni-netd\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219724    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-system-cni-dir\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.219761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219743    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-multus-certs\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219772    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09d024f0-dc43-4c99-81dd-ec9a2c126e86-hosts-file\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219797    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-slash\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:25.219804    4550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219824    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-host\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219848    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-cni-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:25.219860    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls podName:a0bd6859-5176-430c-88cc-507c3f6c136b nodeName:}" failed. No retries permitted until 2026-10-29 21:46:25.71984635 +0000 UTC m=+1.876971859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8bd7" (UID: "a0bd6859-5176-430c-88cc-507c3f6c136b") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219877    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-cnibin\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219891    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-lib-modules\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219904    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-host\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219918    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-etc-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219933    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-netns\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219933    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-ovn\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219946    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-cni-bin\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219971    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-cni-bin\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219976    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-cni-bin\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219902    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-cni-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220006    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-etc-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220019    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-cni-netd\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220027    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-netns\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.219748    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-kubelet\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220061    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-run-multus-certs\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220109    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-cnibin\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220123    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-socket-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220063    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-lib-modules\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220185    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2-host\") pod \"node-ca-rx95w\" (UID: \"a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2\") " pod="openshift-image-registry/node-ca-rx95w"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220195    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-host\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220066    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09d024f0-dc43-4c99-81dd-ec9a2c126e86-hosts-file\") pod \"node-resolver-6bf42\" (UID: \"09d024f0-dc43-4c99-81dd-ec9a2c126e86\") " pod="openshift-dns/node-resolver-6bf42"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220222    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-slash\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220223    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-systemd\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220260    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.220271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220285    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-device-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220310    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220333    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220336    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-systemd-units\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220261    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-systemd\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220369    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220376    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-systemd-units\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220383    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220403    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-var-lib-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220411    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220429    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-device-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220436    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-var-lib-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220472    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-hostroot\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220510    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-conf-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220534    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-hostroot\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220541    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-modprobe-d\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220560    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-conf-dir\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220566    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220598    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-log-socket\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220623    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-socket-dir-parent\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220629    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-modprobe-d\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220646    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-cni-multus\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220654    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-log-socket\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220669    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-kubernetes\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220692    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-cni-multus\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220700    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-multus-socket-dir-parent\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220694    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-run-openvswitch\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220693    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-sys-fs\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220731    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-kubernetes\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220746    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-sys-fs\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220755    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-registration-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220781    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-node-log\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220807    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220832    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-kubelet\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220851    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-node-log\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220854    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-etc-kubernetes\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220869    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b9b32d7-fb57-41c2-8b64-e5e4a2449169-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx2lj\" (UID: \"7b9b32d7-fb57-41c2-8b64-e5e4a2449169\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220883    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12cd2b00-faff-422b-a588-c07050ab96f7-registration-dir\") pod \"aws-ebs-csi-driver-node-kdzfs\" (UID: \"12cd2b00-faff-422b-a588-c07050ab96f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kdzfs"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220891    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-host-var-lib-kubelet\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220899    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3d1ef12-c282-4cba-9678-e369daacc3d1-etc-kubernetes\") pod \"multus-zmrms\" (UID: \"e3d1ef12-c282-4cba-9678-e369daacc3d1\") " pod="openshift-multus/multus-zmrms"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220942    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysconfig\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220984    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cnibin\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.220994    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysconfig\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.221002    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-host-slash\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.221019    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.221034    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf9beffb-08c3-4766-bdd8-cdeaa557e5be-cnibin\") pod \"multus-additional-cni-plugins-4kdn5\" (UID: \"bf9beffb-08c3-4766-bdd8-cdeaa557e5be\") " pod="openshift-multus/multus-additional-cni-plugins-4kdn5"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.221036    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c53ce77-00c9-4509-9650-115117f93d54-kubelet-config\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.221064    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c53ce77-00c9-4509-9650-115117f93d54-kubelet-config\") pod \"global-pull-secret-syncer-7rvg5\" (UID: \"9c53ce77-00c9-4509-9650-115117f93d54\") " pod="kube-system/global-pull-secret-syncer-7rvg5"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.221065    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysctl-d\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.221071    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/193ad4d9-4ef2-4ede-abd6-6ab5b589321a-host-slash\") pod \"iptables-alerter-crv8b\" (UID: \"193ad4d9-4ef2-4ede-abd6-6ab5b589321a\") " pod="openshift-network-operator/iptables-alerter-crv8b"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.221092    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-sys\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.221135    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-sys\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.221168    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/991476c7-c67c-436a-83e0-e1514350d019-etc-sysctl-d\") pod \"tuned-zqvj9\" (UID: \"991476c7-c67c-436a-83e0-e1514350d019\") " pod="openshift-cluster-node-tuning-operator/tuned-zqvj9"
Oct 29 21:46:25.221505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.221405    4550 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 21:46:25.223370 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.223353    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff21346-754f-4de1-b935-26f9a51145fe-metrics-certs\") pod \"network-metrics-daemon-t68qw\" (UID: \"aff21346-754f-4de1-b935-26f9a51145fe\") " pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:25.276659 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.276642    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t68qw"
Oct 29 21:46:25.289169 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.289121    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:25.474364 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.474309    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:25.490167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.490124    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:25.605213 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.605190    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t68qw"]
Oct 29 21:46:25.608373 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:25.608344    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff21346_754f_4de1_b935_26f9a51145fe.slice/crio-d3fff9f66b31c795e2d51dac57f69dfd2a2698f005b1553f7b22b34cd619e0ad WatchSource:0}: Error finding container d3fff9f66b31c795e2d51dac57f69dfd2a2698f005b1553f7b22b34cd619e0ad: Status 404 returned error can't find the container with id d3fff9f66b31c795e2d51dac57f69dfd2a2698f005b1553f7b22b34cd619e0ad
Oct 29 21:46:25.610585 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.610569    4550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 21:46:25.687469 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.687429    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:25.725651 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.725592    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:25.725733 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:25.725722    4550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:25.725801 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:25.725785    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls podName:a0bd6859-5176-430c-88cc-507c3f6c136b nodeName:}" failed. No retries permitted until 2026-10-29 21:46:26.725772555 +0000 UTC m=+2.882898051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8bd7" (UID: "a0bd6859-5176-430c-88cc-507c3f6c136b") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:25.887172 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:25.887127    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:26.056501 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:26.056426    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t68qw" event={"ID":"aff21346-754f-4de1-b935-26f9a51145fe","Type":"ContainerStarted","Data":"d3fff9f66b31c795e2d51dac57f69dfd2a2698f005b1553f7b22b34cd619e0ad"}
Oct 29 21:46:26.085787 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:26.085765    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:26.734694 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:26.734660    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:26.734868 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:26.734794    4550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:26.734868 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:26.734853    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls podName:a0bd6859-5176-430c-88cc-507c3f6c136b nodeName:}" failed. No retries permitted until 2026-10-29 21:46:28.73484035 +0000 UTC m=+4.891965839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8bd7" (UID: "a0bd6859-5176-430c-88cc-507c3f6c136b") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:27.298194 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:27.298091    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:27.415078 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:27.415043    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:27.533849 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:27.533816    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:27.687284 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:27.687256    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:27.889778 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:27.889748    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:28.006745 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:28.006654    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:28.067168 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:28.067123    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t68qw" event={"ID":"aff21346-754f-4de1-b935-26f9a51145fe","Type":"ContainerStarted","Data":"c43584fdd62c614d49f773f5859c338db9b23627a0215f54181fdecb968940f0"}
Oct 29 21:46:28.067294 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:28.067187    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t68qw" event={"ID":"aff21346-754f-4de1-b935-26f9a51145fe","Type":"ContainerStarted","Data":"93d05dae6b6d5bf7a4c82496b47f76524a153236f9ab3ec56a9a10d0798f6d00"}
Oct 29 21:46:28.122901 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:28.122873    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:28.748587 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:28.748547    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:28.749001 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:28.748675    4550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:28.749001 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:28.748731    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls podName:a0bd6859-5176-430c-88cc-507c3f6c136b nodeName:}" failed. No retries permitted until 2026-10-29 21:46:32.748717103 +0000 UTC m=+8.905842580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8bd7" (UID: "a0bd6859-5176-430c-88cc-507c3f6c136b") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:29.286338 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:29.286306    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:29.427707 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:29.427679    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:29.484135 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:29.484109    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:29.686587 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:29.686537    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:29.882564 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:29.882537    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:30.089096 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:30.089072    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:30.207770 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:30.207742    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:30.482836 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:30.482765    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:32.740686 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:32.740651    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:32.772643 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:32.772618    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:32.774774 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:32.774753    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0bd6859-5176-430c-88cc-507c3f6c136b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8bd7\" (UID: \"a0bd6859-5176-430c-88cc-507c3f6c136b\") " pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:32.783726 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:32.783708    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d8bd7"
Oct 29 21:46:33.094551 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:33.094517    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d8bd7"]
Oct 29 21:46:33.098705 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:33.098679    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0bd6859_5176_430c_88cc_507c3f6c136b.slice/crio-f54def0dc117446e3349f9c57f844e64aab62e0de2238f4a5389f30c298a7b0e WatchSource:0}: Error finding container f54def0dc117446e3349f9c57f844e64aab62e0de2238f4a5389f30c298a7b0e: Status 404 returned error can't find the container with id f54def0dc117446e3349f9c57f844e64aab62e0de2238f4a5389f30c298a7b0e
Oct 29 21:46:33.647158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:33.647119    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:34.089227 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:34.089191    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8bd7" event={"ID":"a0bd6859-5176-430c-88cc-507c3f6c136b","Type":"ContainerStarted","Data":"6f3af2df2c0a5185a33bf0f9125ce195d5ce62c5645e3af0bff5030cf2a93770"}
Oct 29 21:46:34.089583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:34.089251    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8bd7" event={"ID":"a0bd6859-5176-430c-88cc-507c3f6c136b","Type":"ContainerStarted","Data":"69935be98c4b640610f8e18200a78897f4b660b4d941a451683311c4fa601a59"}
Oct 29 21:46:34.089583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:34.089275    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8bd7" event={"ID":"a0bd6859-5176-430c-88cc-507c3f6c136b","Type":"ContainerStarted","Data":"f54def0dc117446e3349f9c57f844e64aab62e0de2238f4a5389f30c298a7b0e"}
Oct 29 21:46:34.249960 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:34.249928    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:34.250193 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:34.250179    4550 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:46:35.008294 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:35.008265    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:35.058942 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:35.058918    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x5mwg"
Oct 29 21:46:35.096162 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:35.096119    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8bd7" event={"ID":"a0bd6859-5176-430c-88cc-507c3f6c136b","Type":"ContainerStarted","Data":"221cedaf6be211a4061f76c8d0b326b76e5ea196bb04b38e38c5ff46dcaa2fe9"}
Oct 29 21:46:35.644445 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:35.644415    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:35.800354 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:35.800325    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:36.215739 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:36.215717    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx2lj"
Oct 29 21:46:37.200693 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.200665    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh2g2"]
Oct 29 21:46:37.202513 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.202495    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh2g2"
Oct 29 21:46:37.240185 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.240129    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 29 21:46:37.240691 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.240368    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-df2pq\""
Oct 29 21:46:37.240924 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.240903    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh2g2"]
Oct 29 21:46:37.268410 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.268388    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-cqnpv"]
Oct 29 21:46:37.270315 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.270296    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-cqnpv"
Oct 29 21:46:37.284108 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.284087    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5689cfdf84-bdbdz"]
Oct 29 21:46:37.285751 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.285737    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.286345 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.286325    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 29 21:46:37.286419 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.286328    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 29 21:46:37.297418 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.297398    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-cqnpv"]
Oct 29 21:46:37.302824 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.302805    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9174ad9f-ec68-430f-990d-75c2c26817e1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-dh2g2\" (UID: \"9174ad9f-ec68-430f-990d-75c2c26817e1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh2g2"
Oct 29 21:46:37.319267 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.319244    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 29 21:46:37.319956 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.319936    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5689cfdf84-bdbdz"]
Oct 29 21:46:37.333368 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.333341    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 29 21:46:37.333602 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.333586    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-p5vk4\""
Oct 29 21:46:37.333643 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.333599    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 29 21:46:37.333643 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.333617    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cgft7\""
Oct 29 21:46:37.333722 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.333653    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 29 21:46:37.345056 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.345030    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:37.403615 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.403590    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/526e5ad0-b682-4946-9a79-c83b34eed94b-tmp\") pod \"downloads-6dd789c8b7-cqnpv\" (UID: \"526e5ad0-b682-4946-9a79-c83b34eed94b\") " pod="openshift-console/downloads-6dd789c8b7-cqnpv"
Oct 29 21:46:37.403718 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.403627    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea13b764-00ed-4482-943c-337fb11ae9e5-registry-tls\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.403718 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.403647    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgw4p\" (UniqueName: \"kubernetes.io/projected/ea13b764-00ed-4482-943c-337fb11ae9e5-kube-api-access-pgw4p\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.403718 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.403674    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9174ad9f-ec68-430f-990d-75c2c26817e1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-dh2g2\" (UID: \"9174ad9f-ec68-430f-990d-75c2c26817e1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh2g2"
Oct 29 21:46:37.403855 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.403716    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea13b764-00ed-4482-943c-337fb11ae9e5-installation-pull-secrets\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.403855 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.403787    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbwdq\" (UniqueName: \"kubernetes.io/projected/526e5ad0-b682-4946-9a79-c83b34eed94b-kube-api-access-dbwdq\") pod \"downloads-6dd789c8b7-cqnpv\" (UID: \"526e5ad0-b682-4946-9a79-c83b34eed94b\") " pod="openshift-console/downloads-6dd789c8b7-cqnpv"
Oct 29 21:46:37.403855 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.403804    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea13b764-00ed-4482-943c-337fb11ae9e5-ca-trust-extracted\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.403855 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.403829    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea13b764-00ed-4482-943c-337fb11ae9e5-image-registry-private-configuration\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.403855 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.403852    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea13b764-00ed-4482-943c-337fb11ae9e5-registry-certificates\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.404036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.403929    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea13b764-00ed-4482-943c-337fb11ae9e5-trusted-ca\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.404036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.403951    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea13b764-00ed-4482-943c-337fb11ae9e5-bound-sa-token\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.406176 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.406155    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9174ad9f-ec68-430f-990d-75c2c26817e1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-dh2g2\" (UID: \"9174ad9f-ec68-430f-990d-75c2c26817e1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh2g2"
Oct 29 21:46:37.504878 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.504809    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea13b764-00ed-4482-943c-337fb11ae9e5-installation-pull-secrets\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.504878 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.504854    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbwdq\" (UniqueName: \"kubernetes.io/projected/526e5ad0-b682-4946-9a79-c83b34eed94b-kube-api-access-dbwdq\") pod \"downloads-6dd789c8b7-cqnpv\" (UID: \"526e5ad0-b682-4946-9a79-c83b34eed94b\") " pod="openshift-console/downloads-6dd789c8b7-cqnpv"
Oct 29 21:46:37.505050 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.504880    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea13b764-00ed-4482-943c-337fb11ae9e5-ca-trust-extracted\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.505050 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.504908    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea13b764-00ed-4482-943c-337fb11ae9e5-image-registry-private-configuration\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.505050 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.504932    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea13b764-00ed-4482-943c-337fb11ae9e5-registry-certificates\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.505050 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.504991    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea13b764-00ed-4482-943c-337fb11ae9e5-trusted-ca\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.505050 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.505014    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea13b764-00ed-4482-943c-337fb11ae9e5-bound-sa-token\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.505323 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.505050    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/526e5ad0-b682-4946-9a79-c83b34eed94b-tmp\") pod \"downloads-6dd789c8b7-cqnpv\" (UID: \"526e5ad0-b682-4946-9a79-c83b34eed94b\") " pod="openshift-console/downloads-6dd789c8b7-cqnpv"
Oct 29 21:46:37.505323 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.505081    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea13b764-00ed-4482-943c-337fb11ae9e5-registry-tls\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.505430 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.505408    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea13b764-00ed-4482-943c-337fb11ae9e5-ca-trust-extracted\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.505492 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.505433    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgw4p\" (UniqueName: \"kubernetes.io/projected/ea13b764-00ed-4482-943c-337fb11ae9e5-kube-api-access-pgw4p\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.505705 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.505682    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/526e5ad0-b682-4946-9a79-c83b34eed94b-tmp\") pod \"downloads-6dd789c8b7-cqnpv\" (UID: \"526e5ad0-b682-4946-9a79-c83b34eed94b\") " pod="openshift-console/downloads-6dd789c8b7-cqnpv"
Oct 29 21:46:37.505990 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.505963    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea13b764-00ed-4482-943c-337fb11ae9e5-registry-certificates\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.506169 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.506135    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea13b764-00ed-4482-943c-337fb11ae9e5-trusted-ca\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.507349 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.507323    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea13b764-00ed-4482-943c-337fb11ae9e5-installation-pull-secrets\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.507436 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.507410    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea13b764-00ed-4482-943c-337fb11ae9e5-registry-tls\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.507436 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.507431    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea13b764-00ed-4482-943c-337fb11ae9e5-image-registry-private-configuration\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.512266 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.512245    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh2g2"
Oct 29 21:46:37.528738 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.528717    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea13b764-00ed-4482-943c-337fb11ae9e5-bound-sa-token\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.528925 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.528907    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgw4p\" (UniqueName: \"kubernetes.io/projected/ea13b764-00ed-4482-943c-337fb11ae9e5-kube-api-access-pgw4p\") pod \"image-registry-5689cfdf84-bdbdz\" (UID: \"ea13b764-00ed-4482-943c-337fb11ae9e5\") " pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.529007 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.528989    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbwdq\" (UniqueName: \"kubernetes.io/projected/526e5ad0-b682-4946-9a79-c83b34eed94b-kube-api-access-dbwdq\") pod \"downloads-6dd789c8b7-cqnpv\" (UID: \"526e5ad0-b682-4946-9a79-c83b34eed94b\") " pod="openshift-console/downloads-6dd789c8b7-cqnpv"
Oct 29 21:46:37.544456 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.544429    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:37.580476 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.580451    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-cqnpv"
Oct 29 21:46:37.597097 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.597066    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:37.668941 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.668915    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:37.808973 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.808884    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh2g2"]
Oct 29 21:46:37.811364 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:37.811342    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9174ad9f_ec68_430f_990d_75c2c26817e1.slice/crio-3cecf92467a7fac44115fa161457120821a36e4d0c3c2c96b5d8f57aee388ece WatchSource:0}: Error finding container 3cecf92467a7fac44115fa161457120821a36e4d0c3c2c96b5d8f57aee388ece: Status 404 returned error can't find the container with id 3cecf92467a7fac44115fa161457120821a36e4d0c3c2c96b5d8f57aee388ece
Oct 29 21:46:37.877818 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.877794    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-cqnpv"]
Oct 29 21:46:37.880508 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:37.880481    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526e5ad0_b682_4946_9a79_c83b34eed94b.slice/crio-0f604dea1cd23ec964312c22cb14ddc0c124496ac1d6e43c6a315dabf31602cd WatchSource:0}: Error finding container 0f604dea1cd23ec964312c22cb14ddc0c124496ac1d6e43c6a315dabf31602cd: Status 404 returned error can't find the container with id 0f604dea1cd23ec964312c22cb14ddc0c124496ac1d6e43c6a315dabf31602cd
Oct 29 21:46:37.905599 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:37.905576    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5689cfdf84-bdbdz"]
Oct 29 21:46:37.908233 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:37.908209    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea13b764_00ed_4482_943c_337fb11ae9e5.slice/crio-063196b06a1bd7078aa475a8ec3272f9b792cfefaf04468d50ef2f164b71858f WatchSource:0}: Error finding container 063196b06a1bd7078aa475a8ec3272f9b792cfefaf04468d50ef2f164b71858f: Status 404 returned error can't find the container with id 063196b06a1bd7078aa475a8ec3272f9b792cfefaf04468d50ef2f164b71858f
Oct 29 21:46:38.106051 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:38.106015    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz" event={"ID":"ea13b764-00ed-4482-943c-337fb11ae9e5","Type":"ContainerStarted","Data":"c0b408810d97a5298400706c874cd2891b7f5fab77b6b1b039bb8202d025f2b0"}
Oct 29 21:46:38.106225 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:38.106175    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:38.106225 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:38.106216    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz" event={"ID":"ea13b764-00ed-4482-943c-337fb11ae9e5","Type":"ContainerStarted","Data":"063196b06a1bd7078aa475a8ec3272f9b792cfefaf04468d50ef2f164b71858f"}
Oct 29 21:46:38.107404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:38.107374    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-cqnpv" event={"ID":"526e5ad0-b682-4946-9a79-c83b34eed94b","Type":"ContainerStarted","Data":"0f604dea1cd23ec964312c22cb14ddc0c124496ac1d6e43c6a315dabf31602cd"}
Oct 29 21:46:38.108614 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:38.108583    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh2g2" event={"ID":"9174ad9f-ec68-430f-990d-75c2c26817e1","Type":"ContainerStarted","Data":"3cecf92467a7fac44115fa161457120821a36e4d0c3c2c96b5d8f57aee388ece"}
Oct 29 21:46:38.154042 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:38.153975    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz" podStartSLOduration=1.153955747 podStartE2EDuration="1.153955747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:38.153443296 +0000 UTC m=+14.310568813" watchObservedRunningTime="2026-10-29 21:46:38.153955747 +0000 UTC m=+14.311081250"
Oct 29 21:46:39.114096 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:39.114053    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh2g2" event={"ID":"9174ad9f-ec68-430f-990d-75c2c26817e1","Type":"ContainerStarted","Data":"ecf2dc32727c5e4260dd45d8928e2155fa48c2ac9b53cf504394570303db14c9"}
Oct 29 21:46:39.114612 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:39.114467    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh2g2"
Oct 29 21:46:39.119652 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:39.119630    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh2g2"
Oct 29 21:46:39.156481 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:39.156427    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dh2g2" podStartSLOduration=1.115185677 podStartE2EDuration="2.156414149s" totalImagesPullingTime="1.041228472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:39.156345293 +0000 UTC m=+15.313470810" watchObservedRunningTime="2026-10-29 21:46:39.156414149 +0000 UTC m=+15.313539644"
Oct 29 21:46:39.272177 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:39.272018    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x5mwg_84b63148-f00f-4506-b07d-9c0c0777034c/dns/0.log"
Oct 29 21:46:39.468367 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:39.468296    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x5mwg_84b63148-f00f-4506-b07d-9c0c0777034c/kube-rbac-proxy/0.log"
Oct 29 21:46:39.668597 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:39.668558    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6bf42_09d024f0-dc43-4c99-81dd-ec9a2c126e86/dns-node-resolver/0.log"
Oct 29 21:46:39.906647 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:39.906603    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:40.024887 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.024858    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"]
Oct 29 21:46:40.027156 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.027118    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"
Oct 29 21:46:40.046180 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.046138    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gx6kt\""
Oct 29 21:46:40.058497 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.058470    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 29 21:46:40.059163 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.059123    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:40.059534 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.059515    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"]
Oct 29 21:46:40.062527 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.062505    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 21:46:40.070825 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.070807    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 21:46:40.070927 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.070850    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 21:46:40.124407 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.124382    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqdlt\" (UniqueName: \"kubernetes.io/projected/658d3ab9-05e5-416e-9d80-0ab77a49885b-kube-api-access-cqdlt\") pod \"prometheus-operator-9cd89b74c-x77sz\" (UID: \"658d3ab9-05e5-416e-9d80-0ab77a49885b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"
Oct 29 21:46:40.124796 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.124416    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/658d3ab9-05e5-416e-9d80-0ab77a49885b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-x77sz\" (UID: \"658d3ab9-05e5-416e-9d80-0ab77a49885b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"
Oct 29 21:46:40.124796 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.124434    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/658d3ab9-05e5-416e-9d80-0ab77a49885b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-x77sz\" (UID: \"658d3ab9-05e5-416e-9d80-0ab77a49885b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"
Oct 29 21:46:40.124796 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.124520    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/658d3ab9-05e5-416e-9d80-0ab77a49885b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-x77sz\" (UID: \"658d3ab9-05e5-416e-9d80-0ab77a49885b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"
Oct 29 21:46:40.146926 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.146907    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 29 21:46:40.186812 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.186754    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:40.225479 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.225455    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/658d3ab9-05e5-416e-9d80-0ab77a49885b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-x77sz\" (UID: \"658d3ab9-05e5-416e-9d80-0ab77a49885b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"
Oct 29 21:46:40.225595 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.225545    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdlt\" (UniqueName: \"kubernetes.io/projected/658d3ab9-05e5-416e-9d80-0ab77a49885b-kube-api-access-cqdlt\") pod \"prometheus-operator-9cd89b74c-x77sz\" (UID: \"658d3ab9-05e5-416e-9d80-0ab77a49885b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"
Oct 29 21:46:40.225595 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.225577    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/658d3ab9-05e5-416e-9d80-0ab77a49885b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-x77sz\" (UID: \"658d3ab9-05e5-416e-9d80-0ab77a49885b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"
Oct 29 21:46:40.225705 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.225602    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/658d3ab9-05e5-416e-9d80-0ab77a49885b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-x77sz\" (UID: \"658d3ab9-05e5-416e-9d80-0ab77a49885b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"
Oct 29 21:46:40.226380 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.226358    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/658d3ab9-05e5-416e-9d80-0ab77a49885b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-x77sz\" (UID: \"658d3ab9-05e5-416e-9d80-0ab77a49885b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"
Oct 29 21:46:40.228468 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.228443    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/658d3ab9-05e5-416e-9d80-0ab77a49885b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-x77sz\" (UID: \"658d3ab9-05e5-416e-9d80-0ab77a49885b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"
Oct 29 21:46:40.230592 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.230565    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/658d3ab9-05e5-416e-9d80-0ab77a49885b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-x77sz\" (UID: \"658d3ab9-05e5-416e-9d80-0ab77a49885b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"
Oct 29 21:46:40.250375 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.250351    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqdlt\" (UniqueName: \"kubernetes.io/projected/658d3ab9-05e5-416e-9d80-0ab77a49885b-kube-api-access-cqdlt\") pod \"prometheus-operator-9cd89b74c-x77sz\" (UID: \"658d3ab9-05e5-416e-9d80-0ab77a49885b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"
Oct 29 21:46:40.339330 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.339296    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"
Oct 29 21:46:40.646646 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.646613    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-x77sz"]
Oct 29 21:46:40.649439 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:40.649414    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658d3ab9_05e5_416e_9d80_0ab77a49885b.slice/crio-7cf209d856feff8536a386f436a6e1877c1f3dd86f221e36c459b265593a96ec WatchSource:0}: Error finding container 7cf209d856feff8536a386f436a6e1877c1f3dd86f221e36c459b265593a96ec: Status 404 returned error can't find the container with id 7cf209d856feff8536a386f436a6e1877c1f3dd86f221e36c459b265593a96ec
Oct 29 21:46:40.673531 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:40.673509    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5689cfdf84-bdbdz_ea13b764-00ed-4482-943c-337fb11ae9e5/registry/0.log"
Oct 29 21:46:41.123008 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:41.122969    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz" event={"ID":"658d3ab9-05e5-416e-9d80-0ab77a49885b","Type":"ContainerStarted","Data":"7cf209d856feff8536a386f436a6e1877c1f3dd86f221e36c459b265593a96ec"}
Oct 29 21:46:41.273580 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:41.273553    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rx95w_a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2/node-ca/0.log"
Oct 29 21:46:41.496500 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:41.496415    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:46:41.848343 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:41.848289    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:41.868221 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:41.868198    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sn7rn_ce4f55d6-e263-446d-bb7a-884cc4f1c74c/serve-healthcheck-canary/0.log"
Oct 29 21:46:41.978044 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:41.978012    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:42.133207 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:42.133106    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz" event={"ID":"658d3ab9-05e5-416e-9d80-0ab77a49885b","Type":"ContainerStarted","Data":"4fd57e8e8844c9b6fdbaba12a430583f17024c95eb10d26d43b940a8bf255650"}
Oct 29 21:46:42.133374 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:42.133209    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz" event={"ID":"658d3ab9-05e5-416e-9d80-0ab77a49885b","Type":"ContainerStarted","Data":"d95f61b66e9009e5c6499a51622a4aff10fdf98fda631600b95d40924c249e0b"}
Oct 29 21:46:42.176824 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:42.176768    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x77sz" podStartSLOduration=1.960306426 podStartE2EDuration="3.176752934s" totalImagesPullingTime="1.216446508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:42.176216469 +0000 UTC m=+18.333341965" watchObservedRunningTime="2026-10-29 21:46:42.176752934 +0000 UTC m=+18.333878430"
Oct 29 21:46:42.866218 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:42.866179    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:44.444676 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:44.444644    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:44.730414 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:44.730329    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:44.883786 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:44.883754    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:44.953777 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:44.953748    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sl2vf"]
Oct 29 21:46:44.956771 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:44.956736    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.013160 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.013068    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:45.058859 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.058832    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 21:46:45.059068 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.059052    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 21:46:45.059231 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.059213    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pjw5g\""
Oct 29 21:46:45.059291 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.059213    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 21:46:45.060411 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.060391    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d50eb73-ad36-4be8-90f1-c39479159f97-node-exporter-textfile\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.060520 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.060416    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl7xp\" (UniqueName: \"kubernetes.io/projected/6d50eb73-ad36-4be8-90f1-c39479159f97-kube-api-access-jl7xp\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.060520 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.060439    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d50eb73-ad36-4be8-90f1-c39479159f97-root\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.060520 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.060460    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d50eb73-ad36-4be8-90f1-c39479159f97-node-exporter-tls\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.060520 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.060485    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d50eb73-ad36-4be8-90f1-c39479159f97-metrics-client-ca\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.060690 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.060665    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d50eb73-ad36-4be8-90f1-c39479159f97-node-exporter-wtmp\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.060726 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.060701    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d50eb73-ad36-4be8-90f1-c39479159f97-sys\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.060761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.060737    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d50eb73-ad36-4be8-90f1-c39479159f97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.060811 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.060789    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d50eb73-ad36-4be8-90f1-c39479159f97-node-exporter-accelerators-collector-config\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.161192 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.161122    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d50eb73-ad36-4be8-90f1-c39479159f97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.161366 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.161224    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d50eb73-ad36-4be8-90f1-c39479159f97-node-exporter-accelerators-collector-config\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.161366 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.161254    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d50eb73-ad36-4be8-90f1-c39479159f97-node-exporter-textfile\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.161366 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.161276    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl7xp\" (UniqueName: \"kubernetes.io/projected/6d50eb73-ad36-4be8-90f1-c39479159f97-kube-api-access-jl7xp\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.161366 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.161309    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d50eb73-ad36-4be8-90f1-c39479159f97-root\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.161366 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.161331    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d50eb73-ad36-4be8-90f1-c39479159f97-node-exporter-tls\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.161366 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.161355    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d50eb73-ad36-4be8-90f1-c39479159f97-metrics-client-ca\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.161680 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.161404    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d50eb73-ad36-4be8-90f1-c39479159f97-node-exporter-wtmp\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.161680 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.161432    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d50eb73-ad36-4be8-90f1-c39479159f97-sys\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.161680 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.161510    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d50eb73-ad36-4be8-90f1-c39479159f97-sys\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.161680 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.161564    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d50eb73-ad36-4be8-90f1-c39479159f97-root\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.161979 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.161954    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d50eb73-ad36-4be8-90f1-c39479159f97-node-exporter-textfile\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.162059 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.162006    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d50eb73-ad36-4be8-90f1-c39479159f97-node-exporter-wtmp\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.162205 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.162188    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d50eb73-ad36-4be8-90f1-c39479159f97-node-exporter-accelerators-collector-config\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.162333 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.162305    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d50eb73-ad36-4be8-90f1-c39479159f97-metrics-client-ca\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.163903 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.163880    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d50eb73-ad36-4be8-90f1-c39479159f97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.164029 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.164005    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d50eb73-ad36-4be8-90f1-c39479159f97-node-exporter-tls\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.185099 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.185073    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl7xp\" (UniqueName: \"kubernetes.io/projected/6d50eb73-ad36-4be8-90f1-c39479159f97-kube-api-access-jl7xp\") pod \"node-exporter-sl2vf\" (UID: \"6d50eb73-ad36-4be8-90f1-c39479159f97\") " pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.270796 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.270733    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sl2vf"
Oct 29 21:46:45.282073 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:45.282042    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d50eb73_ad36_4be8_90f1_c39479159f97.slice/crio-f23839cc2c0b6ece24795d0800e6cc058ea616037fa04b5b2ed3d81085539386 WatchSource:0}: Error finding container f23839cc2c0b6ece24795d0800e6cc058ea616037fa04b5b2ed3d81085539386: Status 404 returned error can't find the container with id f23839cc2c0b6ece24795d0800e6cc058ea616037fa04b5b2ed3d81085539386
Oct 29 21:46:45.383606 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.383532    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:45.582748 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.582717    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:45.801426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.801379    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:45.922776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:45.922669    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:46.073527 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:46.073494    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:46.155833 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:46.155787    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sl2vf" event={"ID":"6d50eb73-ad36-4be8-90f1-c39479159f97","Type":"ContainerStarted","Data":"f23839cc2c0b6ece24795d0800e6cc058ea616037fa04b5b2ed3d81085539386"}
Oct 29 21:46:46.275953 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:46.275274    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:46.430611 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:46.430579    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:47.161104 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:47.161069    4550 generic.go:356] "Generic (PLEG): container finished" podID="6d50eb73-ad36-4be8-90f1-c39479159f97" containerID="14505fe629415accc1d9163d44d6db4e9a4ae6ac7254f8c13ffe5ad07a85c561" exitCode=0
Oct 29 21:46:47.161104 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:47.161114    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sl2vf" event={"ID":"6d50eb73-ad36-4be8-90f1-c39479159f97","Type":"ContainerDied","Data":"14505fe629415accc1d9163d44d6db4e9a4ae6ac7254f8c13ffe5ad07a85c561"}
Oct 29 21:46:47.571475 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:47.571396    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:47.694259 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:47.694227    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:48.120469 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:48.120428    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:48.167576 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:48.167533    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sl2vf" event={"ID":"6d50eb73-ad36-4be8-90f1-c39479159f97","Type":"ContainerStarted","Data":"6be26adb2ac0f020c19fc8abe9dda3ff4d5bb12246a5e350f784880656c5a524"}
Oct 29 21:46:48.167974 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:48.167594    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sl2vf" event={"ID":"6d50eb73-ad36-4be8-90f1-c39479159f97","Type":"ContainerStarted","Data":"58a245a5190c63265d631b5ef7c4d457089515d6ac3432a1b231b3d296bf8849"}
Oct 29 21:46:48.226009 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:48.225957    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sl2vf" podStartSLOduration=3.193930461 podStartE2EDuration="4.225943417s" totalImagesPullingTime="952.093691ms" totalInitContainerRuntime="79.919265ms" isStatefulPod=true podCreationTimestamp="2026-10-29 21:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:48.225399636 +0000 UTC m=+24.382525145" watchObservedRunningTime="2026-10-29 21:46:48.225943417 +0000 UTC m=+24.383068959"
Oct 29 21:46:48.242812 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:48.242760    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:48.368651 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:48.368607    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:48.493475 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:48.493374    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:49.495417 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:49.495376    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:49.615494 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:49.615437    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:49.737225 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:49.737186    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:50.644523 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:50.644490    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:50.766323 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:50.766291    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:51.858797 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.858755    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:51.863101 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.863069    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 21:46:51.867214 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.867195    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.900562 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.900532    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 29 21:46:51.900714 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.900646    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 29 21:46:51.900714 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.900689    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 29 21:46:51.900832 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.900816    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 29 21:46:51.900902 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.900891    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 29 21:46:51.901015 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.901001    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 29 21:46:51.901082 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.901069    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 29 21:46:51.901137 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.901107    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 29 21:46:51.901669 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.901645    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 21:46:51.912019 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.911986    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/631c42ab-b3c7-47f2-83ae-8e5798aac342-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912173 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912030    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912173 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912067    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-web-config\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912173 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912108    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-config\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912173 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912154    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912371 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912180    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912371 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912204    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912371 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912230    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912371 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912256    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912371 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912306    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912371 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912337    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912417    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912451    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912480    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97dhd\" (UniqueName: \"kubernetes.io/projected/631c42ab-b3c7-47f2-83ae-8e5798aac342-kube-api-access-97dhd\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912522    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912555    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/631c42ab-b3c7-47f2-83ae-8e5798aac342-config-out\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912581    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912624    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912651    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.912703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.912675    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:51.989458 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.989433    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 29 21:46:51.990622 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.990602    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-18m558d6sbd8a\""
Oct 29 21:46:51.990752 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.990735    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 29 21:46:51.990816 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.990781    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 29 21:46:51.990816 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.990788    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 29 21:46:51.990914 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.990820    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 29 21:46:51.991051 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:51.991030    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9btwq\""
Oct 29 21:46:52.004059 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.004038    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 29 21:46:52.004213 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.004079    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 29 21:46:52.013117 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013083    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013258 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013129    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97dhd\" (UniqueName: \"kubernetes.io/projected/631c42ab-b3c7-47f2-83ae-8e5798aac342-kube-api-access-97dhd\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013258 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013176    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013258 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013213    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/631c42ab-b3c7-47f2-83ae-8e5798aac342-config-out\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013258 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013240    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013447 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013274    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013447 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013298    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013447 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013315    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013447 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013335    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/631c42ab-b3c7-47f2-83ae-8e5798aac342-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013447 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013350    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013447 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013380    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-web-config\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013447 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013418    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-config\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013447 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013442    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013766 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013464    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013766 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013480    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013766 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013502    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013766 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013518    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013766 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013544    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013766 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013567    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.013766 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.013593    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.014567 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.014532    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.016896 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.015376    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.016896 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.015415    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.016896 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.016105    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.016896 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.016570    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.017194 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.017077    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.017765 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.017738    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.019897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.019635    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.020683 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.020313    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.020683 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.020674    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.023292 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.021330    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.023292 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.022154    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-web-config\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.023404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.023352    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.023459 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.023443    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.023630 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.023609    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/631c42ab-b3c7-47f2-83ae-8e5798aac342-config-out\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.023924 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.023903    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.024934 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.024913    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/631c42ab-b3c7-47f2-83ae-8e5798aac342-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.025222 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.025203    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-config\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.032796 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.032773    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.037336 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.037315    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97dhd\" (UniqueName: \"kubernetes.io/projected/631c42ab-b3c7-47f2-83ae-8e5798aac342-kube-api-access-97dhd\") pod \"prometheus-k8s-0\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.195443 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.195367    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:46:52.541443 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:52.541361    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 21:46:52.557921 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:46:52.557890    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631c42ab_b3c7_47f2_83ae_8e5798aac342.slice/crio-dddc4b5127e9b6f178b2748dac424bc853e32e0fac4ceaf006d5f4def5ffe42b WatchSource:0}: Error finding container dddc4b5127e9b6f178b2748dac424bc853e32e0fac4ceaf006d5f4def5ffe42b: Status 404 returned error can't find the container with id dddc4b5127e9b6f178b2748dac424bc853e32e0fac4ceaf006d5f4def5ffe42b
Oct 29 21:46:53.187302 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:53.187259    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"631c42ab-b3c7-47f2-83ae-8e5798aac342","Type":"ContainerStarted","Data":"dddc4b5127e9b6f178b2748dac424bc853e32e0fac4ceaf006d5f4def5ffe42b"}
Oct 29 21:46:53.625548 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:53.625518    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:53.649763 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:53.649737    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hvfh7"
Oct 29 21:46:53.746692 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:53.746623    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:53.955211 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:53.955130    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:54.068181 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:54.068095    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:54.169861 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:46:54.169825    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526e5ad0_b682_4946_9a79_c83b34eed94b.slice/crio-6133a87a06596c82b9e4bf1231c83499c58256ab04c00908543323207875cc96.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 21:46:54.192889 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:54.192857    4550 generic.go:356] "Generic (PLEG): container finished" podID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerID="52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377" exitCode=0
Oct 29 21:46:54.193219 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:54.192940    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"631c42ab-b3c7-47f2-83ae-8e5798aac342","Type":"ContainerDied","Data":"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377"}
Oct 29 21:46:54.711924 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:54.711893    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4hxw6"
Oct 29 21:46:55.199063 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:55.199030    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-cqnpv" event={"ID":"526e5ad0-b682-4946-9a79-c83b34eed94b","Type":"ContainerStarted","Data":"6133a87a06596c82b9e4bf1231c83499c58256ab04c00908543323207875cc96"}
Oct 29 21:46:55.200273 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:55.200248    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-cqnpv"
Oct 29 21:46:55.207867 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:55.207844    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-cqnpv"
Oct 29 21:46:55.290470 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:55.290393    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-cqnpv" podStartSLOduration=2.041643702 podStartE2EDuration="18.290374664s" totalImagesPullingTime="16.248730962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:55.245596589 +0000 UTC m=+31.402722087" watchObservedRunningTime="2026-10-29 21:46:55.290374664 +0000 UTC m=+31.447500162"
Oct 29 21:46:58.324605 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:58.324567    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:58.446816 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:58.446783    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:59.133820 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:59.132521    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5689cfdf84-bdbdz"
Oct 29 21:46:59.263848 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:59.263704    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"631c42ab-b3c7-47f2-83ae-8e5798aac342","Type":"ContainerStarted","Data":"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"}
Oct 29 21:46:59.263848 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:59.263797    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"631c42ab-b3c7-47f2-83ae-8e5798aac342","Type":"ContainerStarted","Data":"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"}
Oct 29 21:46:59.355392 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:59.355344    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:59.477964 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:59.477851    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:59.599381 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:46:59.599346    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:02.280594 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:02.280561    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"631c42ab-b3c7-47f2-83ae-8e5798aac342","Type":"ContainerStarted","Data":"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"}
Oct 29 21:47:02.281030 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:02.280633    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"631c42ab-b3c7-47f2-83ae-8e5798aac342","Type":"ContainerStarted","Data":"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"}
Oct 29 21:47:02.281030 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:02.280675    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"631c42ab-b3c7-47f2-83ae-8e5798aac342","Type":"ContainerStarted","Data":"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"}
Oct 29 21:47:02.281030 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:02.280718    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"631c42ab-b3c7-47f2-83ae-8e5798aac342","Type":"ContainerStarted","Data":"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"}
Oct 29 21:47:02.356554 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:02.356522    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:02.380737 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:02.380690    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.102266719 podStartE2EDuration="11.380672574s" totalImagesPullingTime="8.255400871s" totalInitContainerRuntime="23.004984ms" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:02.367867287 +0000 UTC m=+38.524992805" watchObservedRunningTime="2026-10-29 21:47:02.380672574 +0000 UTC m=+38.537798118"
Oct 29 21:47:02.477844 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:02.477803    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:03.388255 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:03.388202    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:03.510447 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:03.510411    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:04.162590 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:04.162561    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:04.455889 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:04.455818    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:04.575494 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:04.575460    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:05.618259 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:05.618208    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:05.733998 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:05.733968    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:07.196305 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:07.196231    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:09.269709 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:09.269675    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:09.421743 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:09.421713    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:11.488880 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:47:11.488842    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:47:18.834191 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:18.834136    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:18.956405 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:18.956368    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:20.630342 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:20.630287    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:20.751692 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:20.751539    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:28.793750 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:28.793713    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:28.916502 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:28.916470    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:31.359916 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:31.359884    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:31.481784 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:31.481749    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:31.607051 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:31.607018    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:31.677239 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:31.677172    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:31.734809 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:31.734781    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:32.211304 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:32.211278    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:35.616303 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:35.616273    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:35.735397 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:35.735372    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:41.489426 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:47:41.489386    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:47:41.668682 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:41.668639    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:41.784219 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:41.784126    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:42.118366 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:42.118331    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:42.239267 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:42.239236    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:50.628333 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:50.628293    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:50.752655 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:50.752622    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:47:51.496110 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.496066    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 21:47:51.496540 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.496489    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="prometheus" containerID="cri-o://9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c" gracePeriod=600
Oct 29 21:47:51.496769 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.496517    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="kube-rbac-proxy" containerID="cri-o://62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306" gracePeriod=600
Oct 29 21:47:51.496872 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.496523    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="thanos-sidecar" containerID="cri-o://52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb" gracePeriod=600
Oct 29 21:47:51.497106 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.496544    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac" gracePeriod=600
Oct 29 21:47:51.497106 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.496547    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="config-reloader" containerID="cri-o://2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb" gracePeriod=600
Oct 29 21:47:51.497106 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.496560    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="kube-rbac-proxy-web" containerID="cri-o://71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84" gracePeriod=600
Oct 29 21:47:51.748229 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.748177    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:51.896241 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896188    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-0\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.896637 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896292    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-config\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-config\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.896637 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896331    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-db\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.896637 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896523    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-serving-certs-ca-bundle\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.896637 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896570    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.896637 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896613    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-tls\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.896932 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896642    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/631c42ab-b3c7-47f2-83ae-8e5798aac342-kube-api-access-97dhd\" (UniqueName: \"kubernetes.io/projected/631c42ab-b3c7-47f2-83ae-8e5798aac342-kube-api-access-97dhd\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.896932 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896677    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631c42ab-b3c7-47f2-83ae-8e5798aac342-config-out\" (UniqueName: \"kubernetes.io/empty-dir/631c42ab-b3c7-47f2-83ae-8e5798aac342-config-out\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.896932 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896709    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-1\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.896932 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896749    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/631c42ab-b3c7-47f2-83ae-8e5798aac342-tls-assets\" (UniqueName: \"kubernetes.io/projected/631c42ab-b3c7-47f2-83ae-8e5798aac342-tls-assets\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.896932 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896778    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-metrics-client-ca\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.896932 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896812    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-grpc-tls\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.896932 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896853    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-kube-rbac-proxy\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.896932 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896895    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.897349 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896936    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-kubelet-serving-ca-bundle\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.897349 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896947    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-serving-certs-ca-bundle" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:51.897349 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896964    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-thanos-prometheus-http-client-file\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.897349 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.896993    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-metrics-client-certs\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.897349 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.897029    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-2\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.897349 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.897059    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-web-config\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-web-config\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.897349 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.897099    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-trusted-ca-bundle\") pod \"631c42ab-b3c7-47f2-83ae-8e5798aac342\" (UID: \"631c42ab-b3c7-47f2-83ae-8e5798aac342\") "
Oct 29 21:47:51.898026 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.897754    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-db" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:47:51.898161 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.897928    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-kubelet-serving-ca-bundle" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:51.898161 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.898122    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-metrics-client-ca" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:51.898344 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.898315    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-0" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:51.898907 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.898374    4550 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:51.898907 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.898516    4550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-db\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:51.898907 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.898534    4550 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:51.898907 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.898640    4550 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-configmap-metrics-client-ca\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:51.898907 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.898760    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-trusted-ca-bundle" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:51.899275 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.899253    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-1" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:51.899565 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.899538    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631c42ab-b3c7-47f2-83ae-8e5798aac342-tls-assets" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:47:51.899565 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.899554    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-2" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:51.899691 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.899648    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-config" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:51.900598 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.900570    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-tls" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:51.901692 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.901572    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-grpc-tls" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:51.901692 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.901636    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-kube-rbac-proxy" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:51.901846 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.901820    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-thanos-prometheus-http-client-file" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:51.901917 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.901892    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-kube-rbac-proxy-web" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:51.902076 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.902048    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-metrics-client-certs" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:51.902308 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.902289    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-thanos-sidecar-tls" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:51.902534 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.902520    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631c42ab-b3c7-47f2-83ae-8e5798aac342-kube-api-access-97dhd" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "kube-api-access-97dhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:47:51.902595 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.902569    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631c42ab-b3c7-47f2-83ae-8e5798aac342-config-out" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:47:51.913560 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.913531    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-web-config" pod "631c42ab-b3c7-47f2-83ae-8e5798aac342" (UID: "631c42ab-b3c7-47f2-83ae-8e5798aac342"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:51.999907 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.999861    4550 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:51.999907 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.999883    4550 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:51.999907 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.999893    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97dhd\" (UniqueName: \"kubernetes.io/projected/631c42ab-b3c7-47f2-83ae-8e5798aac342-kube-api-access-97dhd\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:51.999907 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.999902    4550 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/631c42ab-b3c7-47f2-83ae-8e5798aac342-config-out\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:52.000075 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.999910    4550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:52.000075 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.999918    4550 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/631c42ab-b3c7-47f2-83ae-8e5798aac342-tls-assets\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:52.000075 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.999929    4550 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-grpc-tls\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:52.000075 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.999945    4550 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-kube-rbac-proxy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:52.000075 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.999953    4550 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:52.000075 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.999963    4550 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:52.000075 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.999972    4550 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-secret-metrics-client-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:52.000075 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.999980    4550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:52.000075 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.999988    4550 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-web-config\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:52.000075 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:51.999996    4550 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:52.000075 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.000003    4550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/631c42ab-b3c7-47f2-83ae-8e5798aac342-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:52.000075 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.000012    4550 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/631c42ab-b3c7-47f2-83ae-8e5798aac342-config\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:47:52.454062 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.454033    4550 generic.go:356] "Generic (PLEG): container finished" podID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerID="f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac" exitCode=0
Oct 29 21:47:52.454062 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.454056    4550 generic.go:356] "Generic (PLEG): container finished" podID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerID="62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306" exitCode=0
Oct 29 21:47:52.454062 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.454063    4550 generic.go:356] "Generic (PLEG): container finished" podID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerID="71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84" exitCode=0
Oct 29 21:47:52.454062 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.454071    4550 generic.go:356] "Generic (PLEG): container finished" podID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerID="52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb" exitCode=0
Oct 29 21:47:52.454356 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.454077    4550 generic.go:356] "Generic (PLEG): container finished" podID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerID="2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb" exitCode=0
Oct 29 21:47:52.454356 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.454082    4550 generic.go:356] "Generic (PLEG): container finished" podID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerID="9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c" exitCode=0
Oct 29 21:47:52.454356 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.454104    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.454356 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.454114    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"631c42ab-b3c7-47f2-83ae-8e5798aac342","Type":"ContainerDied","Data":"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"}
Oct 29 21:47:52.454356 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.454191    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"631c42ab-b3c7-47f2-83ae-8e5798aac342","Type":"ContainerDied","Data":"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"}
Oct 29 21:47:52.454356 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.454218    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"631c42ab-b3c7-47f2-83ae-8e5798aac342","Type":"ContainerDied","Data":"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"}
Oct 29 21:47:52.454356 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.454243    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"631c42ab-b3c7-47f2-83ae-8e5798aac342","Type":"ContainerDied","Data":"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"}
Oct 29 21:47:52.454356 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.454267    4550 scope.go:122] "RemoveContainer" containerID="f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"
Oct 29 21:47:52.454356 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.454269    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"631c42ab-b3c7-47f2-83ae-8e5798aac342","Type":"ContainerDied","Data":"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"}
Oct 29 21:47:52.454680 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.454372    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"631c42ab-b3c7-47f2-83ae-8e5798aac342","Type":"ContainerDied","Data":"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"}
Oct 29 21:47:52.454680 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.454411    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"631c42ab-b3c7-47f2-83ae-8e5798aac342","Type":"ContainerDied","Data":"dddc4b5127e9b6f178b2748dac424bc853e32e0fac4ceaf006d5f4def5ffe42b"}
Oct 29 21:47:52.461679 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.461657    4550 scope.go:122] "RemoveContainer" containerID="62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"
Oct 29 21:47:52.468460 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.468443    4550 scope.go:122] "RemoveContainer" containerID="71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"
Oct 29 21:47:52.474617 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.474600    4550 scope.go:122] "RemoveContainer" containerID="52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"
Oct 29 21:47:52.480682 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.480667    4550 scope.go:122] "RemoveContainer" containerID="2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"
Oct 29 21:47:52.486912 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.486897    4550 scope.go:122] "RemoveContainer" containerID="9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"
Oct 29 21:47:52.493487 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.493471    4550 scope.go:122] "RemoveContainer" containerID="52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377"
Oct 29 21:47:52.499617 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.499438    4550 scope.go:122] "RemoveContainer" containerID="f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"
Oct 29 21:47:52.499754 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:47:52.499735    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac\": container with ID starting with f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac not found: ID does not exist" containerID="f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"
Oct 29 21:47:52.499797 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.499761    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"} err="failed to get container status \"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac\": rpc error: code = NotFound desc = could not find container \"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac\": container with ID starting with f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac not found: ID does not exist"
Oct 29 21:47:52.499797 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.499792    4550 scope.go:122] "RemoveContainer" containerID="62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"
Oct 29 21:47:52.500041 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:47:52.500023    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306\": container with ID starting with 62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306 not found: ID does not exist" containerID="62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"
Oct 29 21:47:52.500095 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.500047    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"} err="failed to get container status \"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306\": rpc error: code = NotFound desc = could not find container \"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306\": container with ID starting with 62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306 not found: ID does not exist"
Oct 29 21:47:52.500095 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.500062    4550 scope.go:122] "RemoveContainer" containerID="71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"
Oct 29 21:47:52.500283 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:47:52.500265    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84\": container with ID starting with 71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84 not found: ID does not exist" containerID="71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"
Oct 29 21:47:52.500326 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.500290    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"} err="failed to get container status \"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84\": rpc error: code = NotFound desc = could not find container \"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84\": container with ID starting with 71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84 not found: ID does not exist"
Oct 29 21:47:52.500326 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.500305    4550 scope.go:122] "RemoveContainer" containerID="52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"
Oct 29 21:47:52.500534 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:47:52.500519    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb\": container with ID starting with 52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb not found: ID does not exist" containerID="52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"
Oct 29 21:47:52.500588 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.500538    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"} err="failed to get container status \"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb\": rpc error: code = NotFound desc = could not find container \"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb\": container with ID starting with 52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb not found: ID does not exist"
Oct 29 21:47:52.500588 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.500551    4550 scope.go:122] "RemoveContainer" containerID="2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"
Oct 29 21:47:52.500794 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:47:52.500776    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb\": container with ID starting with 2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb not found: ID does not exist" containerID="2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"
Oct 29 21:47:52.500835 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.500798    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"} err="failed to get container status \"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb\": rpc error: code = NotFound desc = could not find container \"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb\": container with ID starting with 2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb not found: ID does not exist"
Oct 29 21:47:52.500835 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.500812    4550 scope.go:122] "RemoveContainer" containerID="9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"
Oct 29 21:47:52.501032 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:47:52.501010    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c\": container with ID starting with 9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c not found: ID does not exist" containerID="9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"
Oct 29 21:47:52.501077 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.501031    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"} err="failed to get container status \"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c\": rpc error: code = NotFound desc = could not find container \"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c\": container with ID starting with 9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c not found: ID does not exist"
Oct 29 21:47:52.501077 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.501043    4550 scope.go:122] "RemoveContainer" containerID="52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377"
Oct 29 21:47:52.501519 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:47:52.501503    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377\": container with ID starting with 52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377 not found: ID does not exist" containerID="52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377"
Oct 29 21:47:52.501568 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.501522    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377"} err="failed to get container status \"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377\": rpc error: code = NotFound desc = could not find container \"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377\": container with ID starting with 52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377 not found: ID does not exist"
Oct 29 21:47:52.501568 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.501539    4550 scope.go:122] "RemoveContainer" containerID="f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"
Oct 29 21:47:52.501770 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.501752    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"} err="failed to get container status \"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac\": rpc error: code = NotFound desc = could not find container \"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac\": container with ID starting with f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac not found: ID does not exist"
Oct 29 21:47:52.501812 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.501770    4550 scope.go:122] "RemoveContainer" containerID="62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"
Oct 29 21:47:52.502000 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.501980    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"} err="failed to get container status \"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306\": rpc error: code = NotFound desc = could not find container \"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306\": container with ID starting with 62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306 not found: ID does not exist"
Oct 29 21:47:52.502000 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.501999    4550 scope.go:122] "RemoveContainer" containerID="71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"
Oct 29 21:47:52.502293 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.502276    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"} err="failed to get container status \"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84\": rpc error: code = NotFound desc = could not find container \"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84\": container with ID starting with 71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84 not found: ID does not exist"
Oct 29 21:47:52.502345 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.502293    4550 scope.go:122] "RemoveContainer" containerID="52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"
Oct 29 21:47:52.502502 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.502486    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"} err="failed to get container status \"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb\": rpc error: code = NotFound desc = could not find container \"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb\": container with ID starting with 52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb not found: ID does not exist"
Oct 29 21:47:52.502557 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.502502    4550 scope.go:122] "RemoveContainer" containerID="2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"
Oct 29 21:47:52.502679 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.502666    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"} err="failed to get container status \"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb\": rpc error: code = NotFound desc = could not find container \"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb\": container with ID starting with 2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb not found: ID does not exist"
Oct 29 21:47:52.502727 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.502679    4550 scope.go:122] "RemoveContainer" containerID="9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"
Oct 29 21:47:52.502867 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.502854    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"} err="failed to get container status \"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c\": rpc error: code = NotFound desc = could not find container \"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c\": container with ID starting with 9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c not found: ID does not exist"
Oct 29 21:47:52.502867 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.502866    4550 scope.go:122] "RemoveContainer" containerID="52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377"
Oct 29 21:47:52.503092 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.503073    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377"} err="failed to get container status \"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377\": rpc error: code = NotFound desc = could not find container \"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377\": container with ID starting with 52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377 not found: ID does not exist"
Oct 29 21:47:52.503186 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.503094    4550 scope.go:122] "RemoveContainer" containerID="f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"
Oct 29 21:47:52.503333 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.503317    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"} err="failed to get container status \"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac\": rpc error: code = NotFound desc = could not find container \"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac\": container with ID starting with f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac not found: ID does not exist"
Oct 29 21:47:52.503400 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.503334    4550 scope.go:122] "RemoveContainer" containerID="62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"
Oct 29 21:47:52.503584 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.503564    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"} err="failed to get container status \"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306\": rpc error: code = NotFound desc = could not find container \"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306\": container with ID starting with 62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306 not found: ID does not exist"
Oct 29 21:47:52.503630 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.503584    4550 scope.go:122] "RemoveContainer" containerID="71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"
Oct 29 21:47:52.507664 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.504242    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"} err="failed to get container status \"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84\": rpc error: code = NotFound desc = could not find container \"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84\": container with ID starting with 71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84 not found: ID does not exist"
Oct 29 21:47:52.507664 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.504272    4550 scope.go:122] "RemoveContainer" containerID="52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"
Oct 29 21:47:52.508121 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.508092    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"} err="failed to get container status \"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb\": rpc error: code = NotFound desc = could not find container \"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb\": container with ID starting with 52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb not found: ID does not exist"
Oct 29 21:47:52.508121 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.508120    4550 scope.go:122] "RemoveContainer" containerID="2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"
Oct 29 21:47:52.508385 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.508360    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"} err="failed to get container status \"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb\": rpc error: code = NotFound desc = could not find container \"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb\": container with ID starting with 2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb not found: ID does not exist"
Oct 29 21:47:52.508432 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.508388    4550 scope.go:122] "RemoveContainer" containerID="9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"
Oct 29 21:47:52.508612 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.508592    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"} err="failed to get container status \"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c\": rpc error: code = NotFound desc = could not find container \"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c\": container with ID starting with 9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c not found: ID does not exist"
Oct 29 21:47:52.508663 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.508612    4550 scope.go:122] "RemoveContainer" containerID="52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377"
Oct 29 21:47:52.508886 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.508865    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377"} err="failed to get container status \"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377\": rpc error: code = NotFound desc = could not find container \"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377\": container with ID starting with 52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377 not found: ID does not exist"
Oct 29 21:47:52.508886 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.508885    4550 scope.go:122] "RemoveContainer" containerID="f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"
Oct 29 21:47:52.509129 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.509111    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"} err="failed to get container status \"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac\": rpc error: code = NotFound desc = could not find container \"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac\": container with ID starting with f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac not found: ID does not exist"
Oct 29 21:47:52.509224 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.509130    4550 scope.go:122] "RemoveContainer" containerID="62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"
Oct 29 21:47:52.509428 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.509409    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"} err="failed to get container status \"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306\": rpc error: code = NotFound desc = could not find container \"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306\": container with ID starting with 62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306 not found: ID does not exist"
Oct 29 21:47:52.509428 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.509427    4550 scope.go:122] "RemoveContainer" containerID="71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"
Oct 29 21:47:52.509689 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.509671    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"} err="failed to get container status \"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84\": rpc error: code = NotFound desc = could not find container \"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84\": container with ID starting with 71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84 not found: ID does not exist"
Oct 29 21:47:52.509689 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.509690    4550 scope.go:122] "RemoveContainer" containerID="52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"
Oct 29 21:47:52.509926 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.509907    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"} err="failed to get container status \"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb\": rpc error: code = NotFound desc = could not find container \"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb\": container with ID starting with 52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb not found: ID does not exist"
Oct 29 21:47:52.509965 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.509926    4550 scope.go:122] "RemoveContainer" containerID="2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"
Oct 29 21:47:52.510210 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.510186    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"} err="failed to get container status \"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb\": rpc error: code = NotFound desc = could not find container \"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb\": container with ID starting with 2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb not found: ID does not exist"
Oct 29 21:47:52.510278 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.510211    4550 scope.go:122] "RemoveContainer" containerID="9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"
Oct 29 21:47:52.510470 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.510449    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"} err="failed to get container status \"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c\": rpc error: code = NotFound desc = could not find container \"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c\": container with ID starting with 9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c not found: ID does not exist"
Oct 29 21:47:52.510510 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.510472    4550 scope.go:122] "RemoveContainer" containerID="52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377"
Oct 29 21:47:52.510746 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.510720    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377"} err="failed to get container status \"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377\": rpc error: code = NotFound desc = could not find container \"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377\": container with ID starting with 52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377 not found: ID does not exist"
Oct 29 21:47:52.510746 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.510744    4550 scope.go:122] "RemoveContainer" containerID="f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"
Oct 29 21:47:52.510974 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.510959    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"} err="failed to get container status \"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac\": rpc error: code = NotFound desc = could not find container \"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac\": container with ID starting with f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac not found: ID does not exist"
Oct 29 21:47:52.511018 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.510974    4550 scope.go:122] "RemoveContainer" containerID="62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"
Oct 29 21:47:52.511205 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.511190    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"} err="failed to get container status \"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306\": rpc error: code = NotFound desc = could not find container \"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306\": container with ID starting with 62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306 not found: ID does not exist"
Oct 29 21:47:52.511205 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.511204    4550 scope.go:122] "RemoveContainer" containerID="71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"
Oct 29 21:47:52.511430 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.511411    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"} err="failed to get container status \"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84\": rpc error: code = NotFound desc = could not find container \"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84\": container with ID starting with 71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84 not found: ID does not exist"
Oct 29 21:47:52.511482 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.511431    4550 scope.go:122] "RemoveContainer" containerID="52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"
Oct 29 21:47:52.511631 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.511616    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"} err="failed to get container status \"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb\": rpc error: code = NotFound desc = could not find container \"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb\": container with ID starting with 52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb not found: ID does not exist"
Oct 29 21:47:52.511697 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.511630    4550 scope.go:122] "RemoveContainer" containerID="2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"
Oct 29 21:47:52.511871 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.511851    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"} err="failed to get container status \"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb\": rpc error: code = NotFound desc = could not find container \"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb\": container with ID starting with 2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb not found: ID does not exist"
Oct 29 21:47:52.511954 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.511873    4550 scope.go:122] "RemoveContainer" containerID="9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"
Oct 29 21:47:52.512113 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.512096    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"} err="failed to get container status \"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c\": rpc error: code = NotFound desc = could not find container \"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c\": container with ID starting with 9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c not found: ID does not exist"
Oct 29 21:47:52.512205 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.512114    4550 scope.go:122] "RemoveContainer" containerID="52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377"
Oct 29 21:47:52.512373 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.512357    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377"} err="failed to get container status \"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377\": rpc error: code = NotFound desc = could not find container \"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377\": container with ID starting with 52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377 not found: ID does not exist"
Oct 29 21:47:52.512428 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.512374    4550 scope.go:122] "RemoveContainer" containerID="f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"
Oct 29 21:47:52.512642 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.512622    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac"} err="failed to get container status \"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac\": rpc error: code = NotFound desc = could not find container \"f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac\": container with ID starting with f8f72e136cde431c18d9c326cb4e68289fcbb451e6c23312881af7ef9d11c1ac not found: ID does not exist"
Oct 29 21:47:52.512715 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.512643    4550 scope.go:122] "RemoveContainer" containerID="62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"
Oct 29 21:47:52.512913 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.512893    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306"} err="failed to get container status \"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306\": rpc error: code = NotFound desc = could not find container \"62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306\": container with ID starting with 62c9dd2062a57e901f1d748430367ae95a177ed8986784be38a86036e9a45306 not found: ID does not exist"
Oct 29 21:47:52.512963 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.512914    4550 scope.go:122] "RemoveContainer" containerID="71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"
Oct 29 21:47:52.513204 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.513178    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84"} err="failed to get container status \"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84\": rpc error: code = NotFound desc = could not find container \"71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84\": container with ID starting with 71cf5dc35eb8381b6d7a7ac2438e2647bb6f2b1ba2527de13a60f5139db25a84 not found: ID does not exist"
Oct 29 21:47:52.513259 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.513204    4550 scope.go:122] "RemoveContainer" containerID="52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"
Oct 29 21:47:52.513436 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.513418    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb"} err="failed to get container status \"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb\": rpc error: code = NotFound desc = could not find container \"52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb\": container with ID starting with 52c61948739c253481b59a3e56d0c9e7d357835f058138a2f4b69e7a7422fecb not found: ID does not exist"
Oct 29 21:47:52.513505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.513437    4550 scope.go:122] "RemoveContainer" containerID="2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"
Oct 29 21:47:52.513653 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.513635    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb"} err="failed to get container status \"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb\": rpc error: code = NotFound desc = could not find container \"2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb\": container with ID starting with 2e29522cb51ed0ab69f5d985e8093a8fe5e58b3e94c5e20ad6c6ec044a0f39bb not found: ID does not exist"
Oct 29 21:47:52.513708 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.513653    4550 scope.go:122] "RemoveContainer" containerID="9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"
Oct 29 21:47:52.513877 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.513858    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c"} err="failed to get container status \"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c\": rpc error: code = NotFound desc = could not find container \"9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c\": container with ID starting with 9b10add2d81b20b2e1c3b8fe714b37a42f1e47099cefcc000ff4c80f4b29c94c not found: ID does not exist"
Oct 29 21:47:52.513953 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.513879    4550 scope.go:122] "RemoveContainer" containerID="52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377"
Oct 29 21:47:52.513993 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.513958    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 21:47:52.514157 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.514120    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377"} err="failed to get container status \"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377\": rpc error: code = NotFound desc = could not find container \"52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377\": container with ID starting with 52ab6cb37dc42dcbfb803495569122dae7e609567fe945ac0080f03dbf8ff377 not found: ID does not exist"
Oct 29 21:47:52.525717 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.525697    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 21:47:52.557530 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.557501    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 21:47:52.557796 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.557780    4550 cpu_manager.go:414] "containerMap: removing container" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="init-config-reloader"
Oct 29 21:47:52.557873 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.557800    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="init-config-reloader"
Oct 29 21:47:52.557873 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.557813    4550 cpu_manager.go:414] "containerMap: removing container" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="config-reloader"
Oct 29 21:47:52.557873 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.557820    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="config-reloader"
Oct 29 21:47:52.557873 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.557832    4550 cpu_manager.go:414] "containerMap: removing container" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="thanos-sidecar"
Oct 29 21:47:52.557873 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.557839    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="thanos-sidecar"
Oct 29 21:47:52.557873 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.557849    4550 cpu_manager.go:414] "containerMap: removing container" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="kube-rbac-proxy-thanos"
Oct 29 21:47:52.557873 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.557859    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="kube-rbac-proxy-thanos"
Oct 29 21:47:52.558232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.557884    4550 cpu_manager.go:414] "containerMap: removing container" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="prometheus"
Oct 29 21:47:52.558232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.557891    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="prometheus"
Oct 29 21:47:52.558232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.557899    4550 cpu_manager.go:414] "containerMap: removing container" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="kube-rbac-proxy"
Oct 29 21:47:52.558232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.557906    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="kube-rbac-proxy"
Oct 29 21:47:52.558232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.557932    4550 cpu_manager.go:414] "containerMap: removing container" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="kube-rbac-proxy-web"
Oct 29 21:47:52.558232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.557939    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="kube-rbac-proxy-web"
Oct 29 21:47:52.558232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.558001    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="prometheus"
Oct 29 21:47:52.558232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.558011    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="kube-rbac-proxy"
Oct 29 21:47:52.558232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.558022    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="kube-rbac-proxy-web"
Oct 29 21:47:52.558232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.558030    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="config-reloader"
Oct 29 21:47:52.558232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.558039    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="thanos-sidecar"
Oct 29 21:47:52.558232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.558049    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" containerName="kube-rbac-proxy-thanos"
Oct 29 21:47:52.562064 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.562047    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.579292 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.579270    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 29 21:47:52.591811 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.591681    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 29 21:47:52.591811 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.591699    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 29 21:47:52.591811 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.591743    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 29 21:47:52.591811 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.591763    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 29 21:47:52.591811 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.591777    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 29 21:47:52.592057 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.591824    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 29 21:47:52.592057 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.591836    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 29 21:47:52.592057 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.591859    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 29 21:47:52.592057 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.591893    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 29 21:47:52.594307 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.594286    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-18m558d6sbd8a\""
Oct 29 21:47:52.594307 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.594288    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 29 21:47:52.594427 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.594400    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9btwq\""
Oct 29 21:47:52.595289 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.595267    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 21:47:52.679974 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.679951    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 29 21:47:52.680087 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.679999    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 29 21:47:52.680087 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.680017    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 29 21:47:52.681256 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.681241    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 29 21:47:52.705130 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705080    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705130 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705124    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f07465f-53a1-4caa-8f87-7e1bd060baca-config-out\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705248 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705171    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705292 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705259    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705335 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705288    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705383 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705353    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcfgm\" (UniqueName: \"kubernetes.io/projected/4f07465f-53a1-4caa-8f87-7e1bd060baca-kube-api-access-mcfgm\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705439 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705394    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705494 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705434    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705494 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705459    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705586 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705522    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705586 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705576    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705665 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705604    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f07465f-53a1-4caa-8f87-7e1bd060baca-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705701 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705685    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-config\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705740 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705714    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705796 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705777    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-web-config\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705831 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705817    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4f07465f-53a1-4caa-8f87-7e1bd060baca-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705908 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705864    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705908 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705893    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705977 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705913    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.705977 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.705948    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806374 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806344    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806386    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806412    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f07465f-53a1-4caa-8f87-7e1bd060baca-config-out\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806427    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806444    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806458    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806475    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcfgm\" (UniqueName: \"kubernetes.io/projected/4f07465f-53a1-4caa-8f87-7e1bd060baca-kube-api-access-mcfgm\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806498    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806529    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806552    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806579    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806603    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806624    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f07465f-53a1-4caa-8f87-7e1bd060baca-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806656    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-config\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806680    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806703    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-web-config\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806736    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4f07465f-53a1-4caa-8f87-7e1bd060baca-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.806776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806765    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.807634 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806800    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.807634 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.806834    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.807634 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.807228    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.807634 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.807328    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.807839 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.807688    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.807975 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.807949    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4f07465f-53a1-4caa-8f87-7e1bd060baca-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.809343 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.809318    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f07465f-53a1-4caa-8f87-7e1bd060baca-config-out\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.809653 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.809554    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.809847 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.809826    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.809915 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.809885    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.809976 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.809938    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f07465f-53a1-4caa-8f87-7e1bd060baca-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.810324 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.810300    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-web-config\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.810398 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.810328    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.810881 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.810817    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.811064 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.811044    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.811253 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.811227    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.812103 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.812073    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4f07465f-53a1-4caa-8f87-7e1bd060baca-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.812223 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.812165    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-config\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.812614 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.812594    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.812841 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.812825    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.813061 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.813045    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4f07465f-53a1-4caa-8f87-7e1bd060baca-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.834213 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.834191    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcfgm\" (UniqueName: \"kubernetes.io/projected/4f07465f-53a1-4caa-8f87-7e1bd060baca-kube-api-access-mcfgm\") pod \"prometheus-k8s-0\" (UID: \"4f07465f-53a1-4caa-8f87-7e1bd060baca\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:52.877665 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:52.877642    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:47:53.210477 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:53.210437    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 21:47:53.213374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:47:53.213344    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f07465f_53a1_4caa_8f87_7e1bd060baca.slice/crio-f32b7dccaa08bb888c3ec9642cb2a4dc67adcebfffe02d7045f0155e04857daf WatchSource:0}: Error finding container f32b7dccaa08bb888c3ec9642cb2a4dc67adcebfffe02d7045f0155e04857daf: Status 404 returned error can't find the container with id f32b7dccaa08bb888c3ec9642cb2a4dc67adcebfffe02d7045f0155e04857daf
Oct 29 21:47:53.460017 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:53.459982    4550 generic.go:356] "Generic (PLEG): container finished" podID="4f07465f-53a1-4caa-8f87-7e1bd060baca" containerID="a21d30d6b3232ed0a7783bbcee6a1bf4fc8be439e717edd40b09ded8092ada32" exitCode=0
Oct 29 21:47:53.460137 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:53.460033    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f07465f-53a1-4caa-8f87-7e1bd060baca","Type":"ContainerDied","Data":"a21d30d6b3232ed0a7783bbcee6a1bf4fc8be439e717edd40b09ded8092ada32"}
Oct 29 21:47:53.460137 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:53.460107    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f07465f-53a1-4caa-8f87-7e1bd060baca","Type":"ContainerStarted","Data":"f32b7dccaa08bb888c3ec9642cb2a4dc67adcebfffe02d7045f0155e04857daf"}
Oct 29 21:47:53.986080 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:53.986011    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="631c42ab-b3c7-47f2-83ae-8e5798aac342" path="/var/lib/kubelet/pods/631c42ab-b3c7-47f2-83ae-8e5798aac342/volumes"
Oct 29 21:47:54.466017 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:54.465984    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f07465f-53a1-4caa-8f87-7e1bd060baca","Type":"ContainerStarted","Data":"ed6f9cb365683a3ea99ba5f5c3e145377474c03b35bcbd35d072a30eb4ab18c0"}
Oct 29 21:47:54.466170 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:54.466049    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f07465f-53a1-4caa-8f87-7e1bd060baca","Type":"ContainerStarted","Data":"b4aeb6c029436fa56f79c62c156ec21a3b9a83e33e405d089f2fe934328a07af"}
Oct 29 21:47:54.466170 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:54.466077    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f07465f-53a1-4caa-8f87-7e1bd060baca","Type":"ContainerStarted","Data":"ce4be91a70cb42b365d2cd56a784830441b6ca9551bebbeb09d0ce1c9b749f01"}
Oct 29 21:47:54.466170 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:54.466109    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f07465f-53a1-4caa-8f87-7e1bd060baca","Type":"ContainerStarted","Data":"26987556b978fcfa90c35dfb660cacb9d3853b10e86c0e488115c77cccbb3ce7"}
Oct 29 21:47:54.466170 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:54.466136    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f07465f-53a1-4caa-8f87-7e1bd060baca","Type":"ContainerStarted","Data":"bfeb14e3ac44994ea268525d80b5c9e4e1bdce53e81cd27da111dc2f12208a86"}
Oct 29 21:47:54.466310 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:54.466190    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f07465f-53a1-4caa-8f87-7e1bd060baca","Type":"ContainerStarted","Data":"26d774755c630f871aaa9715852e2e511838ab17521bb8754365ee9cb21cf419"}
Oct 29 21:47:54.537746 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:54.537703    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.518093769 podStartE2EDuration="2.537691007s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.597238ms" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:54.537279102 +0000 UTC m=+90.694404590" watchObservedRunningTime="2026-10-29 21:47:54.537691007 +0000 UTC m=+90.694816558"
Oct 29 21:47:57.877889 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:47:57.877850    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:48:09.392490 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:48:09.392455    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:48:09.868556 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:48:09.868521    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:48:11.503604 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:48:11.503561    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:48:27.460984 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:48:27.460949    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:48:27.772700 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:48:27.772633    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:48:27.894682 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:48:27.894649    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:48:41.489133 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:48:41.489048    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:48:52.877953 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:48:52.877918    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:48:52.893215 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:48:52.893191    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:48:53.678351 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:48:53.678321    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 21:49:09.556598 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:09.556562    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:11.488263 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:49:11.488214    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:49:18.319484 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:18.319452    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:18.451439 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:18.451405    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:18.924361 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:18.924328    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:20.031870 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:20.031836    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:20.249547 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:20.249518    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:20.390128 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:20.390094    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:21.642460 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:21.642422    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:21.792452 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:21.792420    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:23.141318 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:23.141283    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:25.659581 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:25.659548    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:25.778626 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:25.778595    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:26.925275 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:26.925239    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:27.043468 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:27.043440    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:27.166843 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:27.166812    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:30.599654 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:30.599623    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:41.488923 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:49:41.488875    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:49:50.558439 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:50.558400    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:50.734301 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:50.734261    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:51.286320 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.286290    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:51.422857 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.422816    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:51.440559 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.440531    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"]
Oct 29 21:49:51.442739 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.442722    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.482771 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.482741    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cwgd\" (UniqueName: \"kubernetes.io/projected/3ada2eba-d054-4c29-a685-39ef865f2721-kube-api-access-9cwgd\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.482901 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.482805    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ada2eba-d054-4c29-a685-39ef865f2721-manifests\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.482901 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.482872    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ada2eba-d054-4c29-a685-39ef865f2721-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.482988 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.482964    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ada2eba-d054-4c29-a685-39ef865f2721-metrics-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.483031 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.482998    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ada2eba-d054-4c29-a685-39ef865f2721-webhook-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.483067 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.483056    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ada2eba-d054-4c29-a685-39ef865f2721-tmp\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.483739 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.483720    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"]
Oct 29 21:49:51.552830 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.552723    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 29 21:49:51.552960 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.552878    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 29 21:49:51.552960 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.552893    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 29 21:49:51.552960 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.552924    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 29 21:49:51.552960 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.552941    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-6rlhk\""
Oct 29 21:49:51.553098 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.552977    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 29 21:49:51.584041 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.584020    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ada2eba-d054-4c29-a685-39ef865f2721-metrics-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.584430 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.584053    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ada2eba-d054-4c29-a685-39ef865f2721-webhook-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.584430 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.584080    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ada2eba-d054-4c29-a685-39ef865f2721-tmp\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.584430 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.584104    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cwgd\" (UniqueName: \"kubernetes.io/projected/3ada2eba-d054-4c29-a685-39ef865f2721-kube-api-access-9cwgd\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.584430 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.584175    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ada2eba-d054-4c29-a685-39ef865f2721-manifests\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.584430 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.584199    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ada2eba-d054-4c29-a685-39ef865f2721-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.584753 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.584644    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ada2eba-d054-4c29-a685-39ef865f2721-manifests\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.586410 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.586389    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ada2eba-d054-4c29-a685-39ef865f2721-tmp\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.586713 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.586692    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ada2eba-d054-4c29-a685-39ef865f2721-metrics-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.586796 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.586744    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ada2eba-d054-4c29-a685-39ef865f2721-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.586796 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.586757    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ada2eba-d054-4c29-a685-39ef865f2721-webhook-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.610717 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.610696    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cwgd\" (UniqueName: \"kubernetes.io/projected/3ada2eba-d054-4c29-a685-39ef865f2721-kube-api-access-9cwgd\") pod \"opendatahub-operator-controller-manager-59b57f6f78-2lkkd\" (UID: \"3ada2eba-d054-4c29-a685-39ef865f2721\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.715582 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.715548    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:51.755047 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.755023    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:51.843644 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.843614    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:51.978627 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:51.978599    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:52.116196 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:49:52.116169    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ada2eba_d054_4c29_a685_39ef865f2721.slice/crio-a22ae464f3d0706d9279ec9bfba3a19b57c3f8dacba82f7cd49f1772c6c4e4a2 WatchSource:0}: Error finding container a22ae464f3d0706d9279ec9bfba3a19b57c3f8dacba82f7cd49f1772c6c4e4a2: Status 404 returned error can't find the container with id a22ae464f3d0706d9279ec9bfba3a19b57c3f8dacba82f7cd49f1772c6c4e4a2
Oct 29 21:49:52.125789 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:52.125765    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"]
Oct 29 21:49:52.552797 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:52.552725    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:52.681240 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:52.681201    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:52.855400 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:52.855360    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd" event={"ID":"3ada2eba-d054-4c29-a685-39ef865f2721","Type":"ContainerStarted","Data":"a22ae464f3d0706d9279ec9bfba3a19b57c3f8dacba82f7cd49f1772c6c4e4a2"}
Oct 29 21:49:54.864442 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:54.864414    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd" event={"ID":"3ada2eba-d054-4c29-a685-39ef865f2721","Type":"ContainerStarted","Data":"2d2fdb86ee798a22bca2a121c6eeb552f5f827f5a9a44fbbb19025beb71e8057"}
Oct 29 21:49:55.035163 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:55.035116    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:55.163264 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:55.163228    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:55.341620 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:55.341584    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:49:55.869025 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:55.868992    4550 generic.go:356] "Generic (PLEG): container finished" podID="3ada2eba-d054-4c29-a685-39ef865f2721" containerID="2d2fdb86ee798a22bca2a121c6eeb552f5f827f5a9a44fbbb19025beb71e8057" exitCode=0
Oct 29 21:49:55.869412 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:55.869048    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd" event={"ID":"3ada2eba-d054-4c29-a685-39ef865f2721","Type":"ContainerDied","Data":"2d2fdb86ee798a22bca2a121c6eeb552f5f827f5a9a44fbbb19025beb71e8057"}
Oct 29 21:49:56.874579 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:56.874550    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd" event={"ID":"3ada2eba-d054-4c29-a685-39ef865f2721","Type":"ContainerStarted","Data":"b17c3fca6d2529d6922b85a65122f0a16498dde1927c787a9c4bcc460547b898"}
Oct 29 21:49:56.874969 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:56.874660    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:49:56.975082 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:49:56.975032    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd" podStartSLOduration=2.576172501 podStartE2EDuration="5.975018149s" totalImagesPullingTime="2.978683576s" totalInitContainerRuntime="420.162072ms" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:56.962504788 +0000 UTC m=+213.119630285" watchObservedRunningTime="2026-10-29 21:49:56.975018149 +0000 UTC m=+213.132143643"
Oct 29 21:50:00.185779 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:00.185741    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:00.393046 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:00.393005    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:00.599320 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:00.599283    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:00.789043 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:00.789010    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:03.244013 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:03.243951    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:03.476039 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:03.476006    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:03.689742 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:03.689711    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:04.780503 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:04.780463    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:05.008840 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:05.008810    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:05.983370 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:05.983334    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:06.131815 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:06.131781    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:07.880721 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:07.880634    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-2lkkd"
Oct 29 21:50:08.749960 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:08.749927    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:09.576232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:09.576192    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:09.733008 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:09.732971    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:09.950951 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:09.950828    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:10.084407 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.084373    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-srchd"]
Oct 29 21:50:10.088867 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.088847    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd"
Oct 29 21:50:10.113001 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.112973    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-srchd"]
Oct 29 21:50:10.124188 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.124159    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 29 21:50:10.124294 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.124213    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 29 21:50:10.124336 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.124315    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-wvrg5\""
Oct 29 21:50:10.127685 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.127667    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmtpt\" (UniqueName: \"kubernetes.io/projected/93b4187a-1551-4cf8-b807-d057baf03920-kube-api-access-vmtpt\") pod \"cert-manager-webhook-5dbbbf75cc-srchd\" (UID: \"93b4187a-1551-4cf8-b807-d057baf03920\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd"
Oct 29 21:50:10.127738 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.127702    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93b4187a-1551-4cf8-b807-d057baf03920-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-srchd\" (UID: \"93b4187a-1551-4cf8-b807-d057baf03920\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd"
Oct 29 21:50:10.228675 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.228591    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmtpt\" (UniqueName: \"kubernetes.io/projected/93b4187a-1551-4cf8-b807-d057baf03920-kube-api-access-vmtpt\") pod \"cert-manager-webhook-5dbbbf75cc-srchd\" (UID: \"93b4187a-1551-4cf8-b807-d057baf03920\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd"
Oct 29 21:50:10.228675 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.228642    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93b4187a-1551-4cf8-b807-d057baf03920-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-srchd\" (UID: \"93b4187a-1551-4cf8-b807-d057baf03920\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd"
Oct 29 21:50:10.252930 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.252904    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmtpt\" (UniqueName: \"kubernetes.io/projected/93b4187a-1551-4cf8-b807-d057baf03920-kube-api-access-vmtpt\") pod \"cert-manager-webhook-5dbbbf75cc-srchd\" (UID: \"93b4187a-1551-4cf8-b807-d057baf03920\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd"
Oct 29 21:50:10.253071 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.252906    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93b4187a-1551-4cf8-b807-d057baf03920-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-srchd\" (UID: \"93b4187a-1551-4cf8-b807-d057baf03920\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd"
Oct 29 21:50:10.364824 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.364790    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:10.412946 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.412912    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd"
Oct 29 21:50:10.695303 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.695272    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:10.708699 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.708669    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-srchd"]
Oct 29 21:50:10.711552 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:50:10.711522    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93b4187a_1551_4cf8_b807_d057baf03920.slice/crio-ca6460c74e529cbb1ef6c7a3876550e02011c77ccab048af13a72d1607daaded WatchSource:0}: Error finding container ca6460c74e529cbb1ef6c7a3876550e02011c77ccab048af13a72d1607daaded: Status 404 returned error can't find the container with id ca6460c74e529cbb1ef6c7a3876550e02011c77ccab048af13a72d1607daaded
Oct 29 21:50:10.890286 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.890246    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:10.924306 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:10.924272    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" event={"ID":"93b4187a-1551-4cf8-b807-d057baf03920","Type":"ContainerStarted","Data":"ca6460c74e529cbb1ef6c7a3876550e02011c77ccab048af13a72d1607daaded"}
Oct 29 21:50:11.184091 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:11.184060    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:11.355600 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:11.355567    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:11.498487 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:50:11.498410    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:50:11.620549 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:11.620454    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:13.417118 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:13.417073    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:13.937628 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:13.937598    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" event={"ID":"93b4187a-1551-4cf8-b807-d057baf03920","Type":"ContainerStarted","Data":"ef4f6c7b3d5635178d2901210ee471c5359c49ec786acf1ca41971ea3a526948"}
Oct 29 21:50:13.937773 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:13.937715    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd"
Oct 29 21:50:13.980307 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:13.980264    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" podStartSLOduration=0.952248901 podStartE2EDuration="3.980252131s" totalImagesPullingTime="3.02800323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:13.979682386 +0000 UTC m=+230.136808019" watchObservedRunningTime="2026-10-29 21:50:13.980252131 +0000 UTC m=+230.137377628"
Oct 29 21:50:14.064578 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:14.064510    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:14.182106 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:14.182076    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:14.948967 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:14.948933    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-jcqw2"]
Oct 29 21:50:14.951246 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:14.951230    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jcqw2"
Oct 29 21:50:14.978161 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:14.978125    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-sdcxj\""
Oct 29 21:50:14.979297 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:14.979279    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jcqw2"]
Oct 29 21:50:15.074655 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:15.074631    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f1baf9e-ff23-4259-9cbe-9e3445c9e091-bound-sa-token\") pod \"cert-manager-597b95cb84-jcqw2\" (UID: \"8f1baf9e-ff23-4259-9cbe-9e3445c9e091\") " pod="cert-manager/cert-manager-597b95cb84-jcqw2"
Oct 29 21:50:15.074788 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:15.074695    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blxvh\" (UniqueName: \"kubernetes.io/projected/8f1baf9e-ff23-4259-9cbe-9e3445c9e091-kube-api-access-blxvh\") pod \"cert-manager-597b95cb84-jcqw2\" (UID: \"8f1baf9e-ff23-4259-9cbe-9e3445c9e091\") " pod="cert-manager/cert-manager-597b95cb84-jcqw2"
Oct 29 21:50:15.175502 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:15.175472    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f1baf9e-ff23-4259-9cbe-9e3445c9e091-bound-sa-token\") pod \"cert-manager-597b95cb84-jcqw2\" (UID: \"8f1baf9e-ff23-4259-9cbe-9e3445c9e091\") " pod="cert-manager/cert-manager-597b95cb84-jcqw2"
Oct 29 21:50:15.175632 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:15.175544    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blxvh\" (UniqueName: \"kubernetes.io/projected/8f1baf9e-ff23-4259-9cbe-9e3445c9e091-kube-api-access-blxvh\") pod \"cert-manager-597b95cb84-jcqw2\" (UID: \"8f1baf9e-ff23-4259-9cbe-9e3445c9e091\") " pod="cert-manager/cert-manager-597b95cb84-jcqw2"
Oct 29 21:50:15.199362 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:15.199298    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f1baf9e-ff23-4259-9cbe-9e3445c9e091-bound-sa-token\") pod \"cert-manager-597b95cb84-jcqw2\" (UID: \"8f1baf9e-ff23-4259-9cbe-9e3445c9e091\") " pod="cert-manager/cert-manager-597b95cb84-jcqw2"
Oct 29 21:50:15.199577 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:15.199557    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blxvh\" (UniqueName: \"kubernetes.io/projected/8f1baf9e-ff23-4259-9cbe-9e3445c9e091-kube-api-access-blxvh\") pod \"cert-manager-597b95cb84-jcqw2\" (UID: \"8f1baf9e-ff23-4259-9cbe-9e3445c9e091\") " pod="cert-manager/cert-manager-597b95cb84-jcqw2"
Oct 29 21:50:15.260927 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:15.260904    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jcqw2"
Oct 29 21:50:15.559469 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:15.559445    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jcqw2"]
Oct 29 21:50:15.561836 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:50:15.561807    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f1baf9e_ff23_4259_9cbe_9e3445c9e091.slice/crio-fddb323835fd95362486fa789bdeab3c0a4d58475511f3debfede1d83cd74bed WatchSource:0}: Error finding container fddb323835fd95362486fa789bdeab3c0a4d58475511f3debfede1d83cd74bed: Status 404 returned error can't find the container with id fddb323835fd95362486fa789bdeab3c0a4d58475511f3debfede1d83cd74bed
Oct 29 21:50:15.946452 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:15.946413    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jcqw2" event={"ID":"8f1baf9e-ff23-4259-9cbe-9e3445c9e091","Type":"ContainerStarted","Data":"eff0e6a2fccd9db66616f6dec24f7d301c27700a1d81bd96b16abb491965f0b0"}
Oct 29 21:50:15.946625 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:15.946464    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jcqw2" event={"ID":"8f1baf9e-ff23-4259-9cbe-9e3445c9e091","Type":"ContainerStarted","Data":"fddb323835fd95362486fa789bdeab3c0a4d58475511f3debfede1d83cd74bed"}
Oct 29 21:50:15.988348 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:15.988303    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-jcqw2" podStartSLOduration=1.988290745 podStartE2EDuration="1.988290745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:15.987563995 +0000 UTC m=+232.144689493" watchObservedRunningTime="2026-10-29 21:50:15.988290745 +0000 UTC m=+232.145416242"
Oct 29 21:50:19.944178 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:19.944129    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd"
Oct 29 21:50:22.474979 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:22.474948    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:22.680767 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:22.680737    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:22.826731 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:22.826704    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:22.982808 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:22.982772    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:23.121516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.121433    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64545848b5-gp7tk"]
Oct 29 21:50:23.123712 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.123695    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64545848b5-gp7tk"
Oct 29 21:50:23.129333 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.129307    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:23.140312 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.140293    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 29 21:50:23.152546 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.152516    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 29 21:50:23.152697 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.152539    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 29 21:50:23.152697 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.152558    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 29 21:50:23.152826 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.152614    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4n74v\""
Oct 29 21:50:23.153703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.153684    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64545848b5-gp7tk"]
Oct 29 21:50:23.237767 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.237739    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0e0326-d34a-4c53-a2d9-e4fadc814dbc-tls-certs\") pod \"kube-auth-proxy-64545848b5-gp7tk\" (UID: \"9b0e0326-d34a-4c53-a2d9-e4fadc814dbc\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-gp7tk"
Oct 29 21:50:23.237895 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.237785    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b0e0326-d34a-4c53-a2d9-e4fadc814dbc-tmp\") pod \"kube-auth-proxy-64545848b5-gp7tk\" (UID: \"9b0e0326-d34a-4c53-a2d9-e4fadc814dbc\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-gp7tk"
Oct 29 21:50:23.237895 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.237867    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjgk\" (UniqueName: \"kubernetes.io/projected/9b0e0326-d34a-4c53-a2d9-e4fadc814dbc-kube-api-access-pqjgk\") pod \"kube-auth-proxy-64545848b5-gp7tk\" (UID: \"9b0e0326-d34a-4c53-a2d9-e4fadc814dbc\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-gp7tk"
Oct 29 21:50:23.255767 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.255742    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:23.338938 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.338912    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjgk\" (UniqueName: \"kubernetes.io/projected/9b0e0326-d34a-4c53-a2d9-e4fadc814dbc-kube-api-access-pqjgk\") pod \"kube-auth-proxy-64545848b5-gp7tk\" (UID: \"9b0e0326-d34a-4c53-a2d9-e4fadc814dbc\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-gp7tk"
Oct 29 21:50:23.339104 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.338945    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0e0326-d34a-4c53-a2d9-e4fadc814dbc-tls-certs\") pod \"kube-auth-proxy-64545848b5-gp7tk\" (UID: \"9b0e0326-d34a-4c53-a2d9-e4fadc814dbc\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-gp7tk"
Oct 29 21:50:23.339104 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.339054    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b0e0326-d34a-4c53-a2d9-e4fadc814dbc-tmp\") pod \"kube-auth-proxy-64545848b5-gp7tk\" (UID: \"9b0e0326-d34a-4c53-a2d9-e4fadc814dbc\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-gp7tk"
Oct 29 21:50:23.341202 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.341181    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b0e0326-d34a-4c53-a2d9-e4fadc814dbc-tmp\") pod \"kube-auth-proxy-64545848b5-gp7tk\" (UID: \"9b0e0326-d34a-4c53-a2d9-e4fadc814dbc\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-gp7tk"
Oct 29 21:50:23.341426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.341404    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0e0326-d34a-4c53-a2d9-e4fadc814dbc-tls-certs\") pod \"kube-auth-proxy-64545848b5-gp7tk\" (UID: \"9b0e0326-d34a-4c53-a2d9-e4fadc814dbc\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-gp7tk"
Oct 29 21:50:23.362338 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.362317    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjgk\" (UniqueName: \"kubernetes.io/projected/9b0e0326-d34a-4c53-a2d9-e4fadc814dbc-kube-api-access-pqjgk\") pod \"kube-auth-proxy-64545848b5-gp7tk\" (UID: \"9b0e0326-d34a-4c53-a2d9-e4fadc814dbc\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-gp7tk"
Oct 29 21:50:23.390842 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.390783    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:23.434993 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.434970    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64545848b5-gp7tk"
Oct 29 21:50:23.532955 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.532929    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:23.731121 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.731097    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64545848b5-gp7tk"]
Oct 29 21:50:23.733507 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:50:23.733476    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0e0326_d34a_4c53_a2d9_e4fadc814dbc.slice/crio-c23580fe093bcb1a455d704c5926d87eba37ea5b0e9e7a18b1a0531dd0b4cff7 WatchSource:0}: Error finding container c23580fe093bcb1a455d704c5926d87eba37ea5b0e9e7a18b1a0531dd0b4cff7: Status 404 returned error can't find the container with id c23580fe093bcb1a455d704c5926d87eba37ea5b0e9e7a18b1a0531dd0b4cff7
Oct 29 21:50:23.975235 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:23.975205    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64545848b5-gp7tk" event={"ID":"9b0e0326-d34a-4c53-a2d9-e4fadc814dbc","Type":"ContainerStarted","Data":"c23580fe093bcb1a455d704c5926d87eba37ea5b0e9e7a18b1a0531dd0b4cff7"}
Oct 29 21:50:24.909731 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:24.909688    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:25.809731 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:25.809709    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 29 21:50:25.985500 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:25.985470    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64545848b5-gp7tk" event={"ID":"9b0e0326-d34a-4c53-a2d9-e4fadc814dbc","Type":"ContainerStarted","Data":"87d353c84ddfc9d94608508857e9b5fb1084aef4b8aaf43245200519eaa6f44e"}
Oct 29 21:50:26.027547 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:26.027503    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64545848b5-gp7tk" podStartSLOduration=0.969973527 podStartE2EDuration="3.027492209s" totalImagesPullingTime="2.057518682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:26.027285673 +0000 UTC m=+242.184411174" watchObservedRunningTime="2026-10-29 21:50:26.027492209 +0000 UTC m=+242.184617706"
Oct 29 21:50:26.272248 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:26.272208    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:26.408663 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:26.408636    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:27.494703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.494673    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-57f5c66455-wqzgt"]
Oct 29 21:50:27.496955 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.496939    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-wqzgt"
Oct 29 21:50:27.511839 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.511821    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mmtf5\""
Oct 29 21:50:27.524136 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.524114    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 29 21:50:27.524247 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.524186    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 29 21:50:27.524247 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.524221    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 29 21:50:27.525236 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.525218    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-57f5c66455-wqzgt"]
Oct 29 21:50:27.533085 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.533061    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:27.571446 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.571421    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns86c\" (UniqueName: \"kubernetes.io/projected/09c103da-69b2-47c3-a35e-2dc18f6efee7-kube-api-access-ns86c\") pod \"dns-operator-controller-manager-57f5c66455-wqzgt\" (UID: \"09c103da-69b2-47c3-a35e-2dc18f6efee7\") " pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-wqzgt"
Oct 29 21:50:27.672783 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.672757    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns86c\" (UniqueName: \"kubernetes.io/projected/09c103da-69b2-47c3-a35e-2dc18f6efee7-kube-api-access-ns86c\") pod \"dns-operator-controller-manager-57f5c66455-wqzgt\" (UID: \"09c103da-69b2-47c3-a35e-2dc18f6efee7\") " pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-wqzgt"
Oct 29 21:50:27.696187 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.696161    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns86c\" (UniqueName: \"kubernetes.io/projected/09c103da-69b2-47c3-a35e-2dc18f6efee7-kube-api-access-ns86c\") pod \"dns-operator-controller-manager-57f5c66455-wqzgt\" (UID: \"09c103da-69b2-47c3-a35e-2dc18f6efee7\") " pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-wqzgt"
Oct 29 21:50:27.739670 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.739646    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/mcp-gateway-controller-6c98b995c5-kczzc"]
Oct 29 21:50:27.741817 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.741803    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-kczzc"
Oct 29 21:50:27.763934 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.763911    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"mcp-gateway-controller-dockercfg-flgtc\""
Oct 29 21:50:27.765038 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.765007    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/mcp-gateway-controller-6c98b995c5-kczzc"]
Oct 29 21:50:27.808429 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.808407    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-wqzgt"
Oct 29 21:50:27.873671 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.873642    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs76l\" (UniqueName: \"kubernetes.io/projected/c6436350-e6cf-4ef9-a0ac-2cc72b34fb0b-kube-api-access-zs76l\") pod \"mcp-gateway-controller-6c98b995c5-kczzc\" (UID: \"c6436350-e6cf-4ef9-a0ac-2cc72b34fb0b\") " pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-kczzc"
Oct 29 21:50:27.974834 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.974801    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs76l\" (UniqueName: \"kubernetes.io/projected/c6436350-e6cf-4ef9-a0ac-2cc72b34fb0b-kube-api-access-zs76l\") pod \"mcp-gateway-controller-6c98b995c5-kczzc\" (UID: \"c6436350-e6cf-4ef9-a0ac-2cc72b34fb0b\") " pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-kczzc"
Oct 29 21:50:27.998877 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:27.998821    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs76l\" (UniqueName: \"kubernetes.io/projected/c6436350-e6cf-4ef9-a0ac-2cc72b34fb0b-kube-api-access-zs76l\") pod \"mcp-gateway-controller-6c98b995c5-kczzc\" (UID: \"c6436350-e6cf-4ef9-a0ac-2cc72b34fb0b\") " pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-kczzc"
Oct 29 21:50:28.053782 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.053757    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-kczzc"
Oct 29 21:50:28.109269 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:50:28.108845    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c103da_69b2_47c3_a35e_2dc18f6efee7.slice/crio-112917c2fbeaac472e687a9f6b4a93cb789494887c810cdbe4b22f99b67f86cd WatchSource:0}: Error finding container 112917c2fbeaac472e687a9f6b4a93cb789494887c810cdbe4b22f99b67f86cd: Status 404 returned error can't find the container with id 112917c2fbeaac472e687a9f6b4a93cb789494887c810cdbe4b22f99b67f86cd
Oct 29 21:50:28.110748 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.110720    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-57f5c66455-wqzgt"]
Oct 29 21:50:28.347270 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.347246    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/mcp-gateway-controller-6c98b995c5-kczzc"]
Oct 29 21:50:28.349495 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:50:28.349468    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6436350_e6cf_4ef9_a0ac_2cc72b34fb0b.slice/crio-a227886507bae954e594af1b64c07e0754137cff31eed7a7e29c64ab43562956 WatchSource:0}: Error finding container a227886507bae954e594af1b64c07e0754137cff31eed7a7e29c64ab43562956: Status 404 returned error can't find the container with id a227886507bae954e594af1b64c07e0754137cff31eed7a7e29c64ab43562956
Oct 29 21:50:28.353281 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.353259    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-779bff956b-4cdnx"]
Oct 29 21:50:28.355933 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.355914    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-779bff956b-4cdnx"
Oct 29 21:50:28.366676 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.366655    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:28.370739 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.370725    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-5cv2n\""
Oct 29 21:50:28.374442 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.374420    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-779bff956b-4cdnx"]
Oct 29 21:50:28.478687 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.478664    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g76qz\" (UniqueName: \"kubernetes.io/projected/3a9b6803-3648-4ea1-95b4-61a8b1245a7b-kube-api-access-g76qz\") pod \"authorino-operator-779bff956b-4cdnx\" (UID: \"3a9b6803-3648-4ea1-95b4-61a8b1245a7b\") " pod="kuadrant-system/authorino-operator-779bff956b-4cdnx"
Oct 29 21:50:28.579533 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.579505    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g76qz\" (UniqueName: \"kubernetes.io/projected/3a9b6803-3648-4ea1-95b4-61a8b1245a7b-kube-api-access-g76qz\") pod \"authorino-operator-779bff956b-4cdnx\" (UID: \"3a9b6803-3648-4ea1-95b4-61a8b1245a7b\") " pod="kuadrant-system/authorino-operator-779bff956b-4cdnx"
Oct 29 21:50:28.603383 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.603316    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g76qz\" (UniqueName: \"kubernetes.io/projected/3a9b6803-3648-4ea1-95b4-61a8b1245a7b-kube-api-access-g76qz\") pod \"authorino-operator-779bff956b-4cdnx\" (UID: \"3a9b6803-3648-4ea1-95b4-61a8b1245a7b\") " pod="kuadrant-system/authorino-operator-779bff956b-4cdnx"
Oct 29 21:50:28.667729 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.667527    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-779bff956b-4cdnx"
Oct 29 21:50:28.748607 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.748575    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:28.968514 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.968480    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-779bff956b-4cdnx"]
Oct 29 21:50:28.970787 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:50:28.970758    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9b6803_3648_4ea1_95b4_61a8b1245a7b.slice/crio-f3a2d074323fab501f4b72158fe1255370bee78b25f27b753a1d42b6fb3bbf2e WatchSource:0}: Error finding container f3a2d074323fab501f4b72158fe1255370bee78b25f27b753a1d42b6fb3bbf2e: Status 404 returned error can't find the container with id f3a2d074323fab501f4b72158fe1255370bee78b25f27b753a1d42b6fb3bbf2e
Oct 29 21:50:28.996322 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.996293    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-wqzgt" event={"ID":"09c103da-69b2-47c3-a35e-2dc18f6efee7","Type":"ContainerStarted","Data":"112917c2fbeaac472e687a9f6b4a93cb789494887c810cdbe4b22f99b67f86cd"}
Oct 29 21:50:28.997510 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.997480    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-779bff956b-4cdnx" event={"ID":"3a9b6803-3648-4ea1-95b4-61a8b1245a7b","Type":"ContainerStarted","Data":"f3a2d074323fab501f4b72158fe1255370bee78b25f27b753a1d42b6fb3bbf2e"}
Oct 29 21:50:28.998507 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:28.998483    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-kczzc" event={"ID":"c6436350-e6cf-4ef9-a0ac-2cc72b34fb0b","Type":"ContainerStarted","Data":"a227886507bae954e594af1b64c07e0754137cff31eed7a7e29c64ab43562956"}
Oct 29 21:50:29.159760 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:29.159723    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:29.330477 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:29.330446    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:30.513648 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:30.513610    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:30.709793 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:30.709713    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:32.762219 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:32.762187    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:32.885953 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:32.885913    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:33.019123 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:33.019040    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-779bff956b-4cdnx" event={"ID":"3a9b6803-3648-4ea1-95b4-61a8b1245a7b","Type":"ContainerStarted","Data":"e5578e0312afb4d5c6917ae106fce27074770858b2617a6029ff88d4d18fce35"}
Oct 29 21:50:33.019298 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:33.019155    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-779bff956b-4cdnx"
Oct 29 21:50:33.020641 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:33.020620    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-kczzc" event={"ID":"c6436350-e6cf-4ef9-a0ac-2cc72b34fb0b","Type":"ContainerStarted","Data":"94d3dec9b0df3273647a543d07fab7767c4981191446554445ff6042d9459bb5"}
Oct 29 21:50:33.020766 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:33.020670    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-kczzc"
Oct 29 21:50:33.022091 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:33.022069    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-wqzgt" event={"ID":"09c103da-69b2-47c3-a35e-2dc18f6efee7","Type":"ContainerStarted","Data":"a3bf094f62ed7845c9a8e87fe484754963551a5732704eb24791c915f71df32e"}
Oct 29 21:50:33.022176 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:33.022119    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-wqzgt"
Oct 29 21:50:33.067122 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:33.067082    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-779bff956b-4cdnx" podStartSLOduration=1.813458855 podStartE2EDuration="5.067071066s" totalImagesPullingTime="3.253612211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:33.065632706 +0000 UTC m=+249.222758203" watchObservedRunningTime="2026-10-29 21:50:33.067071066 +0000 UTC m=+249.224196589"
Oct 29 21:50:33.112529 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:33.112490    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-wqzgt" podStartSLOduration=1.92025969 podStartE2EDuration="6.112478438s" totalImagesPullingTime="4.192218748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:33.111487621 +0000 UTC m=+249.268613118" watchObservedRunningTime="2026-10-29 21:50:33.112478438 +0000 UTC m=+249.269603914"
Oct 29 21:50:33.157378 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:33.157340    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-kczzc" podStartSLOduration=2.121176202 podStartE2EDuration="6.157330592s" totalImagesPullingTime="4.03615439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:33.155501856 +0000 UTC m=+249.312627356" watchObservedRunningTime="2026-10-29 21:50:33.157330592 +0000 UTC m=+249.314456090"
Oct 29 21:50:40.856321 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:40.856284    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:40.973985 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:40.973955    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:41.489724 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:50:41.489688    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:50:42.874511 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:42.874477    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-77dd99c685-2wwlg"]
Oct 29 21:50:42.882089 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:42.882061    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg"
Oct 29 21:50:42.909128 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:42.909096    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-77dd99c685-2wwlg"]
Oct 29 21:50:42.919845 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:42.919820    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 29 21:50:42.919963 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:42.919933    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8846b\""
Oct 29 21:50:42.924814 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:42.924790    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:43.000723 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:43.000696    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db76s\" (UniqueName: \"kubernetes.io/projected/83759470-f59f-4e76-839d-82bc7910168f-kube-api-access-db76s\") pod \"limitador-limitador-77dd99c685-2wwlg\" (UID: \"83759470-f59f-4e76-839d-82bc7910168f\") " pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg"
Oct 29 21:50:43.000833 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:43.000737    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/83759470-f59f-4e76-839d-82bc7910168f-config-file\") pod \"limitador-limitador-77dd99c685-2wwlg\" (UID: \"83759470-f59f-4e76-839d-82bc7910168f\") " pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg"
Oct 29 21:50:43.007065 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:43.007042    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-77dd99c685-2wwlg"]
Oct 29 21:50:43.101642 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:43.101612    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/83759470-f59f-4e76-839d-82bc7910168f-config-file\") pod \"limitador-limitador-77dd99c685-2wwlg\" (UID: \"83759470-f59f-4e76-839d-82bc7910168f\") " pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg"
Oct 29 21:50:43.101768 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:43.101682    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db76s\" (UniqueName: \"kubernetes.io/projected/83759470-f59f-4e76-839d-82bc7910168f-kube-api-access-db76s\") pod \"limitador-limitador-77dd99c685-2wwlg\" (UID: \"83759470-f59f-4e76-839d-82bc7910168f\") " pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg"
Oct 29 21:50:43.102242 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:43.102224    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/83759470-f59f-4e76-839d-82bc7910168f-config-file\") pod \"limitador-limitador-77dd99c685-2wwlg\" (UID: \"83759470-f59f-4e76-839d-82bc7910168f\") " pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg"
Oct 29 21:50:43.125185 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:43.125126    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db76s\" (UniqueName: \"kubernetes.io/projected/83759470-f59f-4e76-839d-82bc7910168f-kube-api-access-db76s\") pod \"limitador-limitador-77dd99c685-2wwlg\" (UID: \"83759470-f59f-4e76-839d-82bc7910168f\") " pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg"
Oct 29 21:50:43.195044 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:43.195017    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg"
Oct 29 21:50:43.491702 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:43.491680    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-77dd99c685-2wwlg"]
Oct 29 21:50:43.493601 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:50:43.493574    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83759470_f59f_4e76_839d_82bc7910168f.slice/crio-ab929d5320aaaab9bc5d0fa433afdee5158fa45d565799137d058a3c0c72f1c6 WatchSource:0}: Error finding container ab929d5320aaaab9bc5d0fa433afdee5158fa45d565799137d058a3c0c72f1c6: Status 404 returned error can't find the container with id ab929d5320aaaab9bc5d0fa433afdee5158fa45d565799137d058a3c0c72f1c6
Oct 29 21:50:43.616028 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:43.615993    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:43.753011 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:43.752927    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:44.028171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:44.027986    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-kczzc"
Oct 29 21:50:44.028171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:44.028056    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-779bff956b-4cdnx"
Oct 29 21:50:44.028171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:44.028095    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-57f5c66455-wqzgt"
Oct 29 21:50:44.066117 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:44.066090    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg" event={"ID":"83759470-f59f-4e76-839d-82bc7910168f","Type":"ContainerStarted","Data":"ab929d5320aaaab9bc5d0fa433afdee5158fa45d565799137d058a3c0c72f1c6"}
Oct 29 21:50:44.363662 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:44.363628    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74ccc46849-slwk7"]
Oct 29 21:50:44.366719 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:44.366700    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ccc46849-slwk7"
Oct 29 21:50:44.386015 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:44.385986    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-h7r8w\""
Oct 29 21:50:44.386760 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:44.386738    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74ccc46849-slwk7"]
Oct 29 21:50:44.458024 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:44.457991    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:44.514226 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:44.514183    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d27hs\" (UniqueName: \"kubernetes.io/projected/c491abbc-e266-41b3-addf-9240ab21d5d8-kube-api-access-d27hs\") pod \"authorino-74ccc46849-slwk7\" (UID: \"c491abbc-e266-41b3-addf-9240ab21d5d8\") " pod="kuadrant-system/authorino-74ccc46849-slwk7"
Oct 29 21:50:44.571762 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:44.571734    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:50:44.615520 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:44.615446    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d27hs\" (UniqueName: \"kubernetes.io/projected/c491abbc-e266-41b3-addf-9240ab21d5d8-kube-api-access-d27hs\") pod \"authorino-74ccc46849-slwk7\" (UID: \"c491abbc-e266-41b3-addf-9240ab21d5d8\") " pod="kuadrant-system/authorino-74ccc46849-slwk7"
Oct 29 21:50:44.639431 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:44.639404    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d27hs\" (UniqueName: \"kubernetes.io/projected/c491abbc-e266-41b3-addf-9240ab21d5d8-kube-api-access-d27hs\") pod \"authorino-74ccc46849-slwk7\" (UID: \"c491abbc-e266-41b3-addf-9240ab21d5d8\") " pod="kuadrant-system/authorino-74ccc46849-slwk7"
Oct 29 21:50:44.678001 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:44.677971    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ccc46849-slwk7"
Oct 29 21:50:44.981902 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:44.981827    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74ccc46849-slwk7"]
Oct 29 21:50:44.984367 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:50:44.984331    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc491abbc_e266_41b3_addf_9240ab21d5d8.slice/crio-57c22126a755921842d2393ef1b26c8206f58b614eb046eed2d5cd2b23b5cb6a WatchSource:0}: Error finding container 57c22126a755921842d2393ef1b26c8206f58b614eb046eed2d5cd2b23b5cb6a: Status 404 returned error can't find the container with id 57c22126a755921842d2393ef1b26c8206f58b614eb046eed2d5cd2b23b5cb6a
Oct 29 21:50:45.071175 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:45.071117    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ccc46849-slwk7" event={"ID":"c491abbc-e266-41b3-addf-9240ab21d5d8","Type":"ContainerStarted","Data":"57c22126a755921842d2393ef1b26c8206f58b614eb046eed2d5cd2b23b5cb6a"}
Oct 29 21:50:45.072713 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:45.072682    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg" event={"ID":"83759470-f59f-4e76-839d-82bc7910168f","Type":"ContainerStarted","Data":"fe1b67f69736b46a7cfe95526b41f84296e7118adaab1181c75caf5b076010a9"}
Oct 29 21:50:45.072813 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:45.072795    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg"
Oct 29 21:50:45.114283 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:45.114237    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg" podStartSLOduration=1.7938210890000001 podStartE2EDuration="3.114225608s" totalImagesPullingTime="1.320404519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:45.113714445 +0000 UTC m=+261.270839944" watchObservedRunningTime="2026-10-29 21:50:45.114225608 +0000 UTC m=+261.271351106"
Oct 29 21:50:47.093808 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:47.084742    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ccc46849-slwk7" event={"ID":"c491abbc-e266-41b3-addf-9240ab21d5d8","Type":"ContainerStarted","Data":"8abda4c3441bab47e5ef1e5bb85e2426b8e80e9f5a59a6a6f4e9efbc365d179b"}
Oct 29 21:50:47.138954 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:47.138905    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-74ccc46849-slwk7" podStartSLOduration=1.16186552 podStartE2EDuration="3.138891847s" totalImagesPullingTime="1.977026327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:47.138799069 +0000 UTC m=+263.295924567" watchObservedRunningTime="2026-10-29 21:50:47.138891847 +0000 UTC m=+263.296017344"
Oct 29 21:50:49.003498 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:49.003461    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74ccc46849-slwk7"]
Oct 29 21:50:49.091914 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:49.091870    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-74ccc46849-slwk7" podUID="c491abbc-e266-41b3-addf-9240ab21d5d8" containerName="authorino" containerID="cri-o://8abda4c3441bab47e5ef1e5bb85e2426b8e80e9f5a59a6a6f4e9efbc365d179b" gracePeriod=30
Oct 29 21:50:49.329545 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:49.329520    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ccc46849-slwk7"
Oct 29 21:50:49.353722 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:49.353697    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c491abbc-e266-41b3-addf-9240ab21d5d8-kube-api-access-d27hs\" (UniqueName: \"kubernetes.io/projected/c491abbc-e266-41b3-addf-9240ab21d5d8-kube-api-access-d27hs\") pod \"c491abbc-e266-41b3-addf-9240ab21d5d8\" (UID: \"c491abbc-e266-41b3-addf-9240ab21d5d8\") "
Oct 29 21:50:49.355716 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:49.355692    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c491abbc-e266-41b3-addf-9240ab21d5d8-kube-api-access-d27hs" pod "c491abbc-e266-41b3-addf-9240ab21d5d8" (UID: "c491abbc-e266-41b3-addf-9240ab21d5d8"). InnerVolumeSpecName "kube-api-access-d27hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:50:49.454693 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:49.454665    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d27hs\" (UniqueName: \"kubernetes.io/projected/c491abbc-e266-41b3-addf-9240ab21d5d8-kube-api-access-d27hs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:50:50.096694 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:50.096659    4550 generic.go:356] "Generic (PLEG): container finished" podID="c491abbc-e266-41b3-addf-9240ab21d5d8" containerID="8abda4c3441bab47e5ef1e5bb85e2426b8e80e9f5a59a6a6f4e9efbc365d179b" exitCode=0
Oct 29 21:50:50.097196 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:50.096711    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74ccc46849-slwk7"
Oct 29 21:50:50.097196 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:50.096737    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ccc46849-slwk7" event={"ID":"c491abbc-e266-41b3-addf-9240ab21d5d8","Type":"ContainerDied","Data":"8abda4c3441bab47e5ef1e5bb85e2426b8e80e9f5a59a6a6f4e9efbc365d179b"}
Oct 29 21:50:50.097196 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:50.096778    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74ccc46849-slwk7" event={"ID":"c491abbc-e266-41b3-addf-9240ab21d5d8","Type":"ContainerDied","Data":"57c22126a755921842d2393ef1b26c8206f58b614eb046eed2d5cd2b23b5cb6a"}
Oct 29 21:50:50.097196 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:50.096798    4550 scope.go:122] "RemoveContainer" containerID="8abda4c3441bab47e5ef1e5bb85e2426b8e80e9f5a59a6a6f4e9efbc365d179b"
Oct 29 21:50:50.104626 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:50.104607    4550 scope.go:122] "RemoveContainer" containerID="8abda4c3441bab47e5ef1e5bb85e2426b8e80e9f5a59a6a6f4e9efbc365d179b"
Oct 29 21:50:50.104914 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:50:50.104892    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8abda4c3441bab47e5ef1e5bb85e2426b8e80e9f5a59a6a6f4e9efbc365d179b\": container with ID starting with 8abda4c3441bab47e5ef1e5bb85e2426b8e80e9f5a59a6a6f4e9efbc365d179b not found: ID does not exist" containerID="8abda4c3441bab47e5ef1e5bb85e2426b8e80e9f5a59a6a6f4e9efbc365d179b"
Oct 29 21:50:50.105007 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:50.104922    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8abda4c3441bab47e5ef1e5bb85e2426b8e80e9f5a59a6a6f4e9efbc365d179b"} err="failed to get container status \"8abda4c3441bab47e5ef1e5bb85e2426b8e80e9f5a59a6a6f4e9efbc365d179b\": rpc error: code = NotFound desc = could not find container \"8abda4c3441bab47e5ef1e5bb85e2426b8e80e9f5a59a6a6f4e9efbc365d179b\": container with ID starting with 8abda4c3441bab47e5ef1e5bb85e2426b8e80e9f5a59a6a6f4e9efbc365d179b not found: ID does not exist"
Oct 29 21:50:50.147839 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:50.147815    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74ccc46849-slwk7"]
Oct 29 21:50:50.152506 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:50.152484    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-74ccc46849-slwk7"]
Oct 29 21:50:51.986197 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:51.986162    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c491abbc-e266-41b3-addf-9240ab21d5d8" path="/var/lib/kubelet/pods/c491abbc-e266-41b3-addf-9240ab21d5d8/volumes"
Oct 29 21:50:56.078382 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:56.078354    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg"
Oct 29 21:50:56.998718 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:56.998681    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-77dd99c685-2wwlg"]
Oct 29 21:50:56.998934 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:56.998912    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg" podUID="83759470-f59f-4e76-839d-82bc7910168f" containerName="limitador" containerID="cri-o://fe1b67f69736b46a7cfe95526b41f84296e7118adaab1181c75caf5b076010a9" gracePeriod=30
Oct 29 21:50:57.543657 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:57.543629    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg"
Oct 29 21:50:57.614666 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:57.614629    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83759470-f59f-4e76-839d-82bc7910168f-kube-api-access-db76s\" (UniqueName: \"kubernetes.io/projected/83759470-f59f-4e76-839d-82bc7910168f-kube-api-access-db76s\") pod \"83759470-f59f-4e76-839d-82bc7910168f\" (UID: \"83759470-f59f-4e76-839d-82bc7910168f\") "
Oct 29 21:50:57.614828 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:57.614761    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83759470-f59f-4e76-839d-82bc7910168f-config-file\" (UniqueName: \"kubernetes.io/configmap/83759470-f59f-4e76-839d-82bc7910168f-config-file\") pod \"83759470-f59f-4e76-839d-82bc7910168f\" (UID: \"83759470-f59f-4e76-839d-82bc7910168f\") "
Oct 29 21:50:57.615086 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:57.615065    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83759470-f59f-4e76-839d-82bc7910168f-config-file" pod "83759470-f59f-4e76-839d-82bc7910168f" (UID: "83759470-f59f-4e76-839d-82bc7910168f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:50:57.616697 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:57.616677    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83759470-f59f-4e76-839d-82bc7910168f-kube-api-access-db76s" pod "83759470-f59f-4e76-839d-82bc7910168f" (UID: "83759470-f59f-4e76-839d-82bc7910168f"). InnerVolumeSpecName "kube-api-access-db76s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:50:57.716279 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:57.716248    4550 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/83759470-f59f-4e76-839d-82bc7910168f-config-file\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:50:57.716279 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:57.716273    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db76s\" (UniqueName: \"kubernetes.io/projected/83759470-f59f-4e76-839d-82bc7910168f-kube-api-access-db76s\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:50:58.130104 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:58.130073    4550 generic.go:356] "Generic (PLEG): container finished" podID="83759470-f59f-4e76-839d-82bc7910168f" containerID="fe1b67f69736b46a7cfe95526b41f84296e7118adaab1181c75caf5b076010a9" exitCode=0
Oct 29 21:50:58.130299 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:58.130124    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg" event={"ID":"83759470-f59f-4e76-839d-82bc7910168f","Type":"ContainerDied","Data":"fe1b67f69736b46a7cfe95526b41f84296e7118adaab1181c75caf5b076010a9"}
Oct 29 21:50:58.130299 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:58.130181    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg" event={"ID":"83759470-f59f-4e76-839d-82bc7910168f","Type":"ContainerDied","Data":"ab929d5320aaaab9bc5d0fa433afdee5158fa45d565799137d058a3c0c72f1c6"}
Oct 29 21:50:58.130299 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:58.130205    4550 scope.go:122] "RemoveContainer" containerID="fe1b67f69736b46a7cfe95526b41f84296e7118adaab1181c75caf5b076010a9"
Oct 29 21:50:58.130299 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:58.130123    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-77dd99c685-2wwlg"
Oct 29 21:50:58.138117 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:58.138097    4550 scope.go:122] "RemoveContainer" containerID="fe1b67f69736b46a7cfe95526b41f84296e7118adaab1181c75caf5b076010a9"
Oct 29 21:50:58.138414 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:50:58.138396    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1b67f69736b46a7cfe95526b41f84296e7118adaab1181c75caf5b076010a9\": container with ID starting with fe1b67f69736b46a7cfe95526b41f84296e7118adaab1181c75caf5b076010a9 not found: ID does not exist" containerID="fe1b67f69736b46a7cfe95526b41f84296e7118adaab1181c75caf5b076010a9"
Oct 29 21:50:58.138462 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:58.138425    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1b67f69736b46a7cfe95526b41f84296e7118adaab1181c75caf5b076010a9"} err="failed to get container status \"fe1b67f69736b46a7cfe95526b41f84296e7118adaab1181c75caf5b076010a9\": rpc error: code = NotFound desc = could not find container \"fe1b67f69736b46a7cfe95526b41f84296e7118adaab1181c75caf5b076010a9\": container with ID starting with fe1b67f69736b46a7cfe95526b41f84296e7118adaab1181c75caf5b076010a9 not found: ID does not exist"
Oct 29 21:50:58.181856 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:58.181819    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-77dd99c685-2wwlg"]
Oct 29 21:50:58.194064 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:58.194034    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-77dd99c685-2wwlg"]
Oct 29 21:50:59.986517 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:50:59.986484    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83759470-f59f-4e76-839d-82bc7910168f" path="/var/lib/kubelet/pods/83759470-f59f-4e76-839d-82bc7910168f/volumes"
Oct 29 21:51:00.579135 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:00.579106    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:00.698032 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:00.698001    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:01.276585 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.276549    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c"]
Oct 29 21:51:01.276973 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.276892    4550 cpu_manager.go:414] "containerMap: removing container" podUID="83759470-f59f-4e76-839d-82bc7910168f" containerName="limitador"
Oct 29 21:51:01.276973 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.276904    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83759470-f59f-4e76-839d-82bc7910168f" containerName="limitador"
Oct 29 21:51:01.276973 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.276918    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c491abbc-e266-41b3-addf-9240ab21d5d8" containerName="authorino"
Oct 29 21:51:01.276973 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.276923    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c491abbc-e266-41b3-addf-9240ab21d5d8" containerName="authorino"
Oct 29 21:51:01.276973 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.276975    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="83759470-f59f-4e76-839d-82bc7910168f" containerName="limitador"
Oct 29 21:51:01.277137 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.276984    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c491abbc-e266-41b3-addf-9240ab21d5d8" containerName="authorino"
Oct 29 21:51:01.281532 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.281516    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c"
Oct 29 21:51:01.300005 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.299977    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 21:51:01.300593 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.300394    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 21:51:01.300593 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.300548    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nz56h\""
Oct 29 21:51:01.300593 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.300586    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 29 21:51:01.301856 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.301827    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c"]
Oct 29 21:51:01.346240 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.346214    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/877d3ba8-a37b-4317-83a7-81d41b1a7842-data\") pod \"postgres-7cdbf6dc44-2tx5c\" (UID: \"877d3ba8-a37b-4317-83a7-81d41b1a7842\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c"
Oct 29 21:51:01.346358 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.346332    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmw9l\" (UniqueName: \"kubernetes.io/projected/877d3ba8-a37b-4317-83a7-81d41b1a7842-kube-api-access-hmw9l\") pod \"postgres-7cdbf6dc44-2tx5c\" (UID: \"877d3ba8-a37b-4317-83a7-81d41b1a7842\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c"
Oct 29 21:51:01.375286 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.375262    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:01.446856 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.446832    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/877d3ba8-a37b-4317-83a7-81d41b1a7842-data\") pod \"postgres-7cdbf6dc44-2tx5c\" (UID: \"877d3ba8-a37b-4317-83a7-81d41b1a7842\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c"
Oct 29 21:51:01.446942 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.446879    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmw9l\" (UniqueName: \"kubernetes.io/projected/877d3ba8-a37b-4317-83a7-81d41b1a7842-kube-api-access-hmw9l\") pod \"postgres-7cdbf6dc44-2tx5c\" (UID: \"877d3ba8-a37b-4317-83a7-81d41b1a7842\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c"
Oct 29 21:51:01.447227 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.447210    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/877d3ba8-a37b-4317-83a7-81d41b1a7842-data\") pod \"postgres-7cdbf6dc44-2tx5c\" (UID: \"877d3ba8-a37b-4317-83a7-81d41b1a7842\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c"
Oct 29 21:51:01.471084 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.471062    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmw9l\" (UniqueName: \"kubernetes.io/projected/877d3ba8-a37b-4317-83a7-81d41b1a7842-kube-api-access-hmw9l\") pod \"postgres-7cdbf6dc44-2tx5c\" (UID: \"877d3ba8-a37b-4317-83a7-81d41b1a7842\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c"
Oct 29 21:51:01.592802 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.592781    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c"
Oct 29 21:51:01.889541 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:01.889518    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c"]
Oct 29 21:51:01.892158 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:51:01.892114    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877d3ba8_a37b_4317_83a7_81d41b1a7842.slice/crio-a3bfaacb9a68527b503ccec16b57ed031d1d3e7d16ecd648e7f8a34e4f6b60cc WatchSource:0}: Error finding container a3bfaacb9a68527b503ccec16b57ed031d1d3e7d16ecd648e7f8a34e4f6b60cc: Status 404 returned error can't find the container with id a3bfaacb9a68527b503ccec16b57ed031d1d3e7d16ecd648e7f8a34e4f6b60cc
Oct 29 21:51:02.147326 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:02.147243    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c" event={"ID":"877d3ba8-a37b-4317-83a7-81d41b1a7842","Type":"ContainerStarted","Data":"a3bfaacb9a68527b503ccec16b57ed031d1d3e7d16ecd648e7f8a34e4f6b60cc"}
Oct 29 21:51:03.745609 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:03.745574    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:07.178600 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:07.178516    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c" event={"ID":"877d3ba8-a37b-4317-83a7-81d41b1a7842","Type":"ContainerStarted","Data":"886f49fb22c4c340a8d86b4ad1e1c740c7d0d4e6f53d587cce48ee3a94e9d986"}
Oct 29 21:51:07.179035 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:07.178651    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c"
Oct 29 21:51:07.220803 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:07.220751    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c" podStartSLOduration=1.252459877 podStartE2EDuration="6.22073936s" totalImagesPullingTime="4.968279483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:51:07.219964726 +0000 UTC m=+283.377090245" watchObservedRunningTime="2026-10-29 21:51:07.22073936 +0000 UTC m=+283.377864858"
Oct 29 21:51:09.783504 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:09.783469    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:11.489426 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:51:11.489392    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:51:13.210625 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:13.210599    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2tx5c"
Oct 29 21:51:20.421411 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.421381    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:20.509530 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.509491    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7cb76cfbb6-fkcm7"]
Oct 29 21:51:20.515224 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.515197    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7"
Oct 29 21:51:20.532353 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.531876    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8cldw\""
Oct 29 21:51:20.532353 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.532034    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 29 21:51:20.544389 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.544361    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 29 21:51:20.544596 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.544575    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 29 21:51:20.544783 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.544762    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 29 21:51:20.546225 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.546200    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cb76cfbb6-fkcm7"]
Oct 29 21:51:20.578340 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.578301    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:20.615755 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.615730    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq64t\" (UniqueName: \"kubernetes.io/projected/8d517c7c-3570-4570-ad81-26ad57874d6d-kube-api-access-hq64t\") pod \"maas-controller-7cb76cfbb6-fkcm7\" (UID: \"8d517c7c-3570-4570-ad81-26ad57874d6d\") " pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7"
Oct 29 21:51:20.615876 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.615770    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d517c7c-3570-4570-ad81-26ad57874d6d-metrics-certs\") pod \"maas-controller-7cb76cfbb6-fkcm7\" (UID: \"8d517c7c-3570-4570-ad81-26ad57874d6d\") " pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7"
Oct 29 21:51:20.615876 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.615789    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d517c7c-3570-4570-ad81-26ad57874d6d-webhook-cert\") pod \"maas-controller-7cb76cfbb6-fkcm7\" (UID: \"8d517c7c-3570-4570-ad81-26ad57874d6d\") " pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7"
Oct 29 21:51:20.632118 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.632092    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 29 21:51:20.716591 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.716492    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq64t\" (UniqueName: \"kubernetes.io/projected/8d517c7c-3570-4570-ad81-26ad57874d6d-kube-api-access-hq64t\") pod \"maas-controller-7cb76cfbb6-fkcm7\" (UID: \"8d517c7c-3570-4570-ad81-26ad57874d6d\") " pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7"
Oct 29 21:51:20.716591 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.716550    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d517c7c-3570-4570-ad81-26ad57874d6d-metrics-certs\") pod \"maas-controller-7cb76cfbb6-fkcm7\" (UID: \"8d517c7c-3570-4570-ad81-26ad57874d6d\") " pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7"
Oct 29 21:51:20.716591 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.716571    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d517c7c-3570-4570-ad81-26ad57874d6d-webhook-cert\") pod \"maas-controller-7cb76cfbb6-fkcm7\" (UID: \"8d517c7c-3570-4570-ad81-26ad57874d6d\") " pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7"
Oct 29 21:51:20.719108 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.719082    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d517c7c-3570-4570-ad81-26ad57874d6d-webhook-cert\") pod \"maas-controller-7cb76cfbb6-fkcm7\" (UID: \"8d517c7c-3570-4570-ad81-26ad57874d6d\") " pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7"
Oct 29 21:51:20.719235 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.719112    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d517c7c-3570-4570-ad81-26ad57874d6d-metrics-certs\") pod \"maas-controller-7cb76cfbb6-fkcm7\" (UID: \"8d517c7c-3570-4570-ad81-26ad57874d6d\") " pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7"
Oct 29 21:51:20.740583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.740553    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:20.760239 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.760213    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq64t\" (UniqueName: \"kubernetes.io/projected/8d517c7c-3570-4570-ad81-26ad57874d6d-kube-api-access-hq64t\") pod \"maas-controller-7cb76cfbb6-fkcm7\" (UID: \"8d517c7c-3570-4570-ad81-26ad57874d6d\") " pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7"
Oct 29 21:51:20.839732 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.839698    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7"
Oct 29 21:51:20.874399 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:20.874364    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:21.047641 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.047577    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f6b55dfc4-hv5c5"]
Oct 29 21:51:21.055110 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.055077    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5"
Oct 29 21:51:21.073433 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.073409    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f6b55dfc4-hv5c5"]
Oct 29 21:51:21.120296 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.120272    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a47b6ed-0ade-4b53-ab33-43e52c37ed92-webhook-cert\") pod \"maas-controller-6f6b55dfc4-hv5c5\" (UID: \"6a47b6ed-0ade-4b53-ab33-43e52c37ed92\") " pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5"
Oct 29 21:51:21.120439 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.120303    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a47b6ed-0ade-4b53-ab33-43e52c37ed92-metrics-certs\") pod \"maas-controller-6f6b55dfc4-hv5c5\" (UID: \"6a47b6ed-0ade-4b53-ab33-43e52c37ed92\") " pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5"
Oct 29 21:51:21.120439 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.120350    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7nfn\" (UniqueName: \"kubernetes.io/projected/6a47b6ed-0ade-4b53-ab33-43e52c37ed92-kube-api-access-b7nfn\") pod \"maas-controller-6f6b55dfc4-hv5c5\" (UID: \"6a47b6ed-0ade-4b53-ab33-43e52c37ed92\") " pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5"
Oct 29 21:51:21.141912 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.141886    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cb76cfbb6-fkcm7"]
Oct 29 21:51:21.144709 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:51:21.144686    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d517c7c_3570_4570_ad81_26ad57874d6d.slice/crio-c63b6a76a12434577ebcbf15b90b10e5e98fa29cf1ac6c816cf568565f2800c3 WatchSource:0}: Error finding container c63b6a76a12434577ebcbf15b90b10e5e98fa29cf1ac6c816cf568565f2800c3: Status 404 returned error can't find the container with id c63b6a76a12434577ebcbf15b90b10e5e98fa29cf1ac6c816cf568565f2800c3
Oct 29 21:51:21.221739 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.221710    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a47b6ed-0ade-4b53-ab33-43e52c37ed92-webhook-cert\") pod \"maas-controller-6f6b55dfc4-hv5c5\" (UID: \"6a47b6ed-0ade-4b53-ab33-43e52c37ed92\") " pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5"
Oct 29 21:51:21.221845 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.221743    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a47b6ed-0ade-4b53-ab33-43e52c37ed92-metrics-certs\") pod \"maas-controller-6f6b55dfc4-hv5c5\" (UID: \"6a47b6ed-0ade-4b53-ab33-43e52c37ed92\") " pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5"
Oct 29 21:51:21.221845 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.221806    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7nfn\" (UniqueName: \"kubernetes.io/projected/6a47b6ed-0ade-4b53-ab33-43e52c37ed92-kube-api-access-b7nfn\") pod \"maas-controller-6f6b55dfc4-hv5c5\" (UID: \"6a47b6ed-0ade-4b53-ab33-43e52c37ed92\") " pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5"
Oct 29 21:51:21.224194 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.224173    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a47b6ed-0ade-4b53-ab33-43e52c37ed92-webhook-cert\") pod \"maas-controller-6f6b55dfc4-hv5c5\" (UID: \"6a47b6ed-0ade-4b53-ab33-43e52c37ed92\") " pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5"
Oct 29 21:51:21.224282 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.224198    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a47b6ed-0ade-4b53-ab33-43e52c37ed92-metrics-certs\") pod \"maas-controller-6f6b55dfc4-hv5c5\" (UID: \"6a47b6ed-0ade-4b53-ab33-43e52c37ed92\") " pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5"
Oct 29 21:51:21.233718 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.233691    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7" event={"ID":"8d517c7c-3570-4570-ad81-26ad57874d6d","Type":"ContainerStarted","Data":"c63b6a76a12434577ebcbf15b90b10e5e98fa29cf1ac6c816cf568565f2800c3"}
Oct 29 21:51:21.245378 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.245359    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7nfn\" (UniqueName: \"kubernetes.io/projected/6a47b6ed-0ade-4b53-ab33-43e52c37ed92-kube-api-access-b7nfn\") pod \"maas-controller-6f6b55dfc4-hv5c5\" (UID: \"6a47b6ed-0ade-4b53-ab33-43e52c37ed92\") " pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5"
Oct 29 21:51:21.369060 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.369030    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5"
Oct 29 21:51:21.679336 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:21.679260    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f6b55dfc4-hv5c5"]
Oct 29 21:51:21.684097 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:51:21.684053    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a47b6ed_0ade_4b53_ab33_43e52c37ed92.slice/crio-1edebf55340db2896f35a80dd107627bd72dd0d2c34bfc6b4b0ac000916caa11 WatchSource:0}: Error finding container 1edebf55340db2896f35a80dd107627bd72dd0d2c34bfc6b4b0ac000916caa11: Status 404 returned error can't find the container with id 1edebf55340db2896f35a80dd107627bd72dd0d2c34bfc6b4b0ac000916caa11
Oct 29 21:51:22.241506 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:22.241468    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5" event={"ID":"6a47b6ed-0ade-4b53-ab33-43e52c37ed92","Type":"ContainerStarted","Data":"1edebf55340db2896f35a80dd107627bd72dd0d2c34bfc6b4b0ac000916caa11"}
Oct 29 21:51:24.173110 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:24.173023    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:24.252994 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:24.252954    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7" event={"ID":"8d517c7c-3570-4570-ad81-26ad57874d6d","Type":"ContainerStarted","Data":"a07dfabb34c5cdf70cd2820f464749301d2d89890e9830deaf13f7ccd9ebdf54"}
Oct 29 21:51:24.253204 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:24.253109    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7"
Oct 29 21:51:24.298883 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:24.298832    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7" podStartSLOduration=1.554413302 podStartE2EDuration="4.298819448s" totalImagesPullingTime="2.744406146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:51:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:51:24.298412535 +0000 UTC m=+300.455538033" watchObservedRunningTime="2026-10-29 21:51:24.298819448 +0000 UTC m=+300.455944946"
Oct 29 21:51:24.316376 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:24.316351    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:25.259054 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:25.259012    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5" event={"ID":"6a47b6ed-0ade-4b53-ab33-43e52c37ed92","Type":"ContainerStarted","Data":"e98de4c27118d469979b57295e0eb402f3e2b2424ca75ee0b8527d44d3a5f407"}
Oct 29 21:51:25.259499 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:25.259336    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5"
Oct 29 21:51:25.312573 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:25.312508    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5" podStartSLOduration=1.656075617 podStartE2EDuration="4.312493209s" totalImagesPullingTime="2.656417592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:51:25.311865535 +0000 UTC m=+301.468991033" watchObservedRunningTime="2026-10-29 21:51:25.312493209 +0000 UTC m=+301.469618744"
Oct 29 21:51:27.428504 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.428470    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"]
Oct 29 21:51:27.432070 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.432048    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:27.453962 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.453938    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 29 21:51:27.454713 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.454690    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"]
Oct 29 21:51:27.465531 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.465511    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 29 21:51:27.465709 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.465555    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 29 21:51:27.465709 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.465644    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-fhvng\""
Oct 29 21:51:27.526076 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.526040    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:27.589088 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.589058    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a11c33aa-7a9a-422f-8697-a36ef4bc30ba-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r8ft2\" (UID: \"a11c33aa-7a9a-422f-8697-a36ef4bc30ba\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:27.589088 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.589094    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a11c33aa-7a9a-422f-8697-a36ef4bc30ba-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r8ft2\" (UID: \"a11c33aa-7a9a-422f-8697-a36ef4bc30ba\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:27.589298 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.589126    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a11c33aa-7a9a-422f-8697-a36ef4bc30ba-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r8ft2\" (UID: \"a11c33aa-7a9a-422f-8697-a36ef4bc30ba\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:27.589298 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.589277    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbgrr\" (UniqueName: \"kubernetes.io/projected/a11c33aa-7a9a-422f-8697-a36ef4bc30ba-kube-api-access-tbgrr\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r8ft2\" (UID: \"a11c33aa-7a9a-422f-8697-a36ef4bc30ba\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:27.662098 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.662055    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:27.690218 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.690120    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a11c33aa-7a9a-422f-8697-a36ef4bc30ba-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r8ft2\" (UID: \"a11c33aa-7a9a-422f-8697-a36ef4bc30ba\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:27.690218 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.690179    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a11c33aa-7a9a-422f-8697-a36ef4bc30ba-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r8ft2\" (UID: \"a11c33aa-7a9a-422f-8697-a36ef4bc30ba\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:27.690218 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.690215    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a11c33aa-7a9a-422f-8697-a36ef4bc30ba-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r8ft2\" (UID: \"a11c33aa-7a9a-422f-8697-a36ef4bc30ba\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:27.690498 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.690254    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbgrr\" (UniqueName: \"kubernetes.io/projected/a11c33aa-7a9a-422f-8697-a36ef4bc30ba-kube-api-access-tbgrr\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r8ft2\" (UID: \"a11c33aa-7a9a-422f-8697-a36ef4bc30ba\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:27.691068 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.691036    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a11c33aa-7a9a-422f-8697-a36ef4bc30ba-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r8ft2\" (UID: \"a11c33aa-7a9a-422f-8697-a36ef4bc30ba\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:27.691234 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.691215    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a11c33aa-7a9a-422f-8697-a36ef4bc30ba-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r8ft2\" (UID: \"a11c33aa-7a9a-422f-8697-a36ef4bc30ba\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:27.692680 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.692653    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a11c33aa-7a9a-422f-8697-a36ef4bc30ba-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r8ft2\" (UID: \"a11c33aa-7a9a-422f-8697-a36ef4bc30ba\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:27.723804 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.723777    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbgrr\" (UniqueName: \"kubernetes.io/projected/a11c33aa-7a9a-422f-8697-a36ef4bc30ba-kube-api-access-tbgrr\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r8ft2\" (UID: \"a11c33aa-7a9a-422f-8697-a36ef4bc30ba\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:27.746654 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:27.746629    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:28.057397 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.057324    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"]
Oct 29 21:51:28.060426 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:51:28.060397    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11c33aa_7a9a_422f_8697_a36ef4bc30ba.slice/crio-2131a2e2a818bfa0be0309f16deaed57f165a604c110ff181646c48e0906e4c2 WatchSource:0}: Error finding container 2131a2e2a818bfa0be0309f16deaed57f165a604c110ff181646c48e0906e4c2: Status 404 returned error can't find the container with id 2131a2e2a818bfa0be0309f16deaed57f165a604c110ff181646c48e0906e4c2
Oct 29 21:51:28.062945 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.062929    4550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 21:51:28.215319 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.215291    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:28.273790 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.273758    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2" event={"ID":"a11c33aa-7a9a-422f-8697-a36ef4bc30ba","Type":"ContainerStarted","Data":"2131a2e2a818bfa0be0309f16deaed57f165a604c110ff181646c48e0906e4c2"}
Oct 29 21:51:28.345529 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.345493    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:28.465677 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.465645    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:28.506947 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.506912    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-f5cm2"]
Oct 29 21:51:28.513926 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.513900    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-f5cm2"
Oct 29 21:51:28.529260 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.529237    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rwhd6\""
Oct 29 21:51:28.529369 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.529341    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-f5cm2"]
Oct 29 21:51:28.540353 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.540332    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 29 21:51:28.549051 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.549028    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"]
Oct 29 21:51:28.552523 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.552500    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"
Oct 29 21:51:28.572815 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.572720    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"]
Oct 29 21:51:28.591389 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.591358    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:28.592069 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.592046    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xcv5v"]
Oct 29 21:51:28.595902 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.595851    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-xcv5v"
Oct 29 21:51:28.617113 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.617091    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xcv5v"]
Oct 29 21:51:28.627063 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.627044    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-8mllk\""
Oct 29 21:51:28.627202 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.627163    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-p9hct\""
Oct 29 21:51:28.627260 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.627215    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 29 21:51:28.627260 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.627250    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 29 21:51:28.639728 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.639702    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-8jrxh"]
Oct 29 21:51:28.643006 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.642989    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-8jrxh"
Oct 29 21:51:28.659574 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.659554    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-lzrxx\""
Oct 29 21:51:28.659661 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.659618    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 29 21:51:28.660661 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.660645    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 29 21:51:28.685449 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.685422    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8jrxh"]
Oct 29 21:51:28.700474 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.700447    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20102b85-6195-42e4-94ea-873d5af3ad35-cert\") pod \"kserve-controller-manager-776cc66d47-f5cm2\" (UID: \"20102b85-6195-42e4-94ea-873d5af3ad35\") " pod="opendatahub/kserve-controller-manager-776cc66d47-f5cm2"
Oct 29 21:51:28.700571 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.700501    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjghb\" (UniqueName: \"kubernetes.io/projected/e3059381-b63e-4d76-bf2d-ade2250ff36a-kube-api-access-cjghb\") pod \"model-serving-api-75bd89fdfc-xcv5v\" (UID: \"e3059381-b63e-4d76-bf2d-ade2250ff36a\") " pod="opendatahub/model-serving-api-75bd89fdfc-xcv5v"
Oct 29 21:51:28.700632 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.700593    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68ff145f-d0fc-42af-8688-f25544f2809d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4xfjh\" (UID: \"68ff145f-d0fc-42af-8688-f25544f2809d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"
Oct 29 21:51:28.700684 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.700650    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnb7z\" (UniqueName: \"kubernetes.io/projected/20102b85-6195-42e4-94ea-873d5af3ad35-kube-api-access-cnb7z\") pod \"kserve-controller-manager-776cc66d47-f5cm2\" (UID: \"20102b85-6195-42e4-94ea-873d5af3ad35\") " pod="opendatahub/kserve-controller-manager-776cc66d47-f5cm2"
Oct 29 21:51:28.700729 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.700714    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3059381-b63e-4d76-bf2d-ade2250ff36a-tls-certs\") pod \"model-serving-api-75bd89fdfc-xcv5v\" (UID: \"e3059381-b63e-4d76-bf2d-ade2250ff36a\") " pod="opendatahub/model-serving-api-75bd89fdfc-xcv5v"
Oct 29 21:51:28.700780 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.700762    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj6xp\" (UniqueName: \"kubernetes.io/projected/68ff145f-d0fc-42af-8688-f25544f2809d-kube-api-access-dj6xp\") pod \"llmisvc-controller-manager-869d5dcc5d-4xfjh\" (UID: \"68ff145f-d0fc-42af-8688-f25544f2809d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"
Oct 29 21:51:28.740030 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.740001    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:28.801313 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.801287    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68ff145f-d0fc-42af-8688-f25544f2809d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4xfjh\" (UID: \"68ff145f-d0fc-42af-8688-f25544f2809d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"
Oct 29 21:51:28.801426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.801334    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnb7z\" (UniqueName: \"kubernetes.io/projected/20102b85-6195-42e4-94ea-873d5af3ad35-kube-api-access-cnb7z\") pod \"kserve-controller-manager-776cc66d47-f5cm2\" (UID: \"20102b85-6195-42e4-94ea-873d5af3ad35\") " pod="opendatahub/kserve-controller-manager-776cc66d47-f5cm2"
Oct 29 21:51:28.801426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.801360    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3059381-b63e-4d76-bf2d-ade2250ff36a-tls-certs\") pod \"model-serving-api-75bd89fdfc-xcv5v\" (UID: \"e3059381-b63e-4d76-bf2d-ade2250ff36a\") " pod="opendatahub/model-serving-api-75bd89fdfc-xcv5v"
Oct 29 21:51:28.801426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.801383    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-cert\") pod \"odh-model-controller-769875db56-8jrxh\" (UID: \"b10e7c81-07e3-4c64-8433-ee31a9a26fbb\") " pod="opendatahub/odh-model-controller-769875db56-8jrxh"
Oct 29 21:51:28.801426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.801406    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-metrics-certs\") pod \"odh-model-controller-769875db56-8jrxh\" (UID: \"b10e7c81-07e3-4c64-8433-ee31a9a26fbb\") " pod="opendatahub/odh-model-controller-769875db56-8jrxh"
Oct 29 21:51:28.801582 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.801425    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj6xp\" (UniqueName: \"kubernetes.io/projected/68ff145f-d0fc-42af-8688-f25544f2809d-kube-api-access-dj6xp\") pod \"llmisvc-controller-manager-869d5dcc5d-4xfjh\" (UID: \"68ff145f-d0fc-42af-8688-f25544f2809d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"
Oct 29 21:51:28.801684 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.801661    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djjlh\" (UniqueName: \"kubernetes.io/projected/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-kube-api-access-djjlh\") pod \"odh-model-controller-769875db56-8jrxh\" (UID: \"b10e7c81-07e3-4c64-8433-ee31a9a26fbb\") " pod="opendatahub/odh-model-controller-769875db56-8jrxh"
Oct 29 21:51:28.801767 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.801751    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20102b85-6195-42e4-94ea-873d5af3ad35-cert\") pod \"kserve-controller-manager-776cc66d47-f5cm2\" (UID: \"20102b85-6195-42e4-94ea-873d5af3ad35\") " pod="opendatahub/kserve-controller-manager-776cc66d47-f5cm2"
Oct 29 21:51:28.801809 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.801790    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjghb\" (UniqueName: \"kubernetes.io/projected/e3059381-b63e-4d76-bf2d-ade2250ff36a-kube-api-access-cjghb\") pod \"model-serving-api-75bd89fdfc-xcv5v\" (UID: \"e3059381-b63e-4d76-bf2d-ade2250ff36a\") " pod="opendatahub/model-serving-api-75bd89fdfc-xcv5v"
Oct 29 21:51:28.803987 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.803965    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68ff145f-d0fc-42af-8688-f25544f2809d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4xfjh\" (UID: \"68ff145f-d0fc-42af-8688-f25544f2809d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"
Oct 29 21:51:28.803987 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.803977    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3059381-b63e-4d76-bf2d-ade2250ff36a-tls-certs\") pod \"model-serving-api-75bd89fdfc-xcv5v\" (UID: \"e3059381-b63e-4d76-bf2d-ade2250ff36a\") " pod="opendatahub/model-serving-api-75bd89fdfc-xcv5v"
Oct 29 21:51:28.804115 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.804100    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20102b85-6195-42e4-94ea-873d5af3ad35-cert\") pod \"kserve-controller-manager-776cc66d47-f5cm2\" (UID: \"20102b85-6195-42e4-94ea-873d5af3ad35\") " pod="opendatahub/kserve-controller-manager-776cc66d47-f5cm2"
Oct 29 21:51:28.825020 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.824997    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj6xp\" (UniqueName: \"kubernetes.io/projected/68ff145f-d0fc-42af-8688-f25544f2809d-kube-api-access-dj6xp\") pod \"llmisvc-controller-manager-869d5dcc5d-4xfjh\" (UID: \"68ff145f-d0fc-42af-8688-f25544f2809d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"
Oct 29 21:51:28.825174 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.825021    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjghb\" (UniqueName: \"kubernetes.io/projected/e3059381-b63e-4d76-bf2d-ade2250ff36a-kube-api-access-cjghb\") pod \"model-serving-api-75bd89fdfc-xcv5v\" (UID: \"e3059381-b63e-4d76-bf2d-ade2250ff36a\") " pod="opendatahub/model-serving-api-75bd89fdfc-xcv5v"
Oct 29 21:51:28.825174 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.825097    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnb7z\" (UniqueName: \"kubernetes.io/projected/20102b85-6195-42e4-94ea-873d5af3ad35-kube-api-access-cnb7z\") pod \"kserve-controller-manager-776cc66d47-f5cm2\" (UID: \"20102b85-6195-42e4-94ea-873d5af3ad35\") " pod="opendatahub/kserve-controller-manager-776cc66d47-f5cm2"
Oct 29 21:51:28.828871 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.828849    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-f5cm2"
Oct 29 21:51:28.865016 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.864956    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"
Oct 29 21:51:28.869986 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.869952    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:28.902985 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.902944    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djjlh\" (UniqueName: \"kubernetes.io/projected/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-kube-api-access-djjlh\") pod \"odh-model-controller-769875db56-8jrxh\" (UID: \"b10e7c81-07e3-4c64-8433-ee31a9a26fbb\") " pod="opendatahub/odh-model-controller-769875db56-8jrxh"
Oct 29 21:51:28.903119 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.903100    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-cert\") pod \"odh-model-controller-769875db56-8jrxh\" (UID: \"b10e7c81-07e3-4c64-8433-ee31a9a26fbb\") " pod="opendatahub/odh-model-controller-769875db56-8jrxh"
Oct 29 21:51:28.903191 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.903164    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-metrics-certs\") pod \"odh-model-controller-769875db56-8jrxh\" (UID: \"b10e7c81-07e3-4c64-8433-ee31a9a26fbb\") " pod="opendatahub/odh-model-controller-769875db56-8jrxh"
Oct 29 21:51:28.905940 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.905772    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-metrics-certs\") pod \"odh-model-controller-769875db56-8jrxh\" (UID: \"b10e7c81-07e3-4c64-8433-ee31a9a26fbb\") " pod="opendatahub/odh-model-controller-769875db56-8jrxh"
Oct 29 21:51:28.905940 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.905901    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-cert\") pod \"odh-model-controller-769875db56-8jrxh\" (UID: \"b10e7c81-07e3-4c64-8433-ee31a9a26fbb\") " pod="opendatahub/odh-model-controller-769875db56-8jrxh"
Oct 29 21:51:28.908803 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.908787    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-xcv5v"
Oct 29 21:51:28.932186 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.931306    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djjlh\" (UniqueName: \"kubernetes.io/projected/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-kube-api-access-djjlh\") pod \"odh-model-controller-769875db56-8jrxh\" (UID: \"b10e7c81-07e3-4c64-8433-ee31a9a26fbb\") " pod="opendatahub/odh-model-controller-769875db56-8jrxh"
Oct 29 21:51:28.955633 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:28.955338    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-8jrxh"
Oct 29 21:51:29.060918 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:29.060880    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:29.128452 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:29.128423    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-f5cm2"]
Oct 29 21:51:29.131006 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:51:29.130962    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20102b85_6195_42e4_94ea_873d5af3ad35.slice/crio-939a2ba6908161e3db7db29a57570111067d93bc22910d5df7355a619865e545 WatchSource:0}: Error finding container 939a2ba6908161e3db7db29a57570111067d93bc22910d5df7355a619865e545: Status 404 returned error can't find the container with id 939a2ba6908161e3db7db29a57570111067d93bc22910d5df7355a619865e545
Oct 29 21:51:29.176603 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:29.176576    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"]
Oct 29 21:51:29.178685 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:51:29.178653    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod68ff145f_d0fc_42af_8688_f25544f2809d.slice/crio-98af217304e99660b1479bbf97badf6f78214e138bb311f32bebd9005b29e3c6 WatchSource:0}: Error finding container 98af217304e99660b1479bbf97badf6f78214e138bb311f32bebd9005b29e3c6: Status 404 returned error can't find the container with id 98af217304e99660b1479bbf97badf6f78214e138bb311f32bebd9005b29e3c6
Oct 29 21:51:29.218260 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:29.218213    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xcv5v"]
Oct 29 21:51:29.222212 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:51:29.221921    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3059381_b63e_4d76_bf2d_ade2250ff36a.slice/crio-fe29cf9aade8798bb82db3ba1fa91a24f2f3e07516dd1089160bfaf003e7778a WatchSource:0}: Error finding container fe29cf9aade8798bb82db3ba1fa91a24f2f3e07516dd1089160bfaf003e7778a: Status 404 returned error can't find the container with id fe29cf9aade8798bb82db3ba1fa91a24f2f3e07516dd1089160bfaf003e7778a
Oct 29 21:51:29.261438 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:29.261400    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8jrxh"]
Oct 29 21:51:29.268663 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:51:29.268638    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10e7c81_07e3_4c64_8433_ee31a9a26fbb.slice/crio-1501df7ed26c06b9a6a271474e463813bda7cd5c85b6402cf8f862345691a7a7 WatchSource:0}: Error finding container 1501df7ed26c06b9a6a271474e463813bda7cd5c85b6402cf8f862345691a7a7: Status 404 returned error can't find the container with id 1501df7ed26c06b9a6a271474e463813bda7cd5c85b6402cf8f862345691a7a7
Oct 29 21:51:29.282796 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:29.282713    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-f5cm2" event={"ID":"20102b85-6195-42e4-94ea-873d5af3ad35","Type":"ContainerStarted","Data":"939a2ba6908161e3db7db29a57570111067d93bc22910d5df7355a619865e545"}
Oct 29 21:51:29.285288 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:29.285254    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-8jrxh" event={"ID":"b10e7c81-07e3-4c64-8433-ee31a9a26fbb","Type":"ContainerStarted","Data":"1501df7ed26c06b9a6a271474e463813bda7cd5c85b6402cf8f862345691a7a7"}
Oct 29 21:51:29.286898 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:29.286873    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-xcv5v" event={"ID":"e3059381-b63e-4d76-bf2d-ade2250ff36a","Type":"ContainerStarted","Data":"fe29cf9aade8798bb82db3ba1fa91a24f2f3e07516dd1089160bfaf003e7778a"}
Oct 29 21:51:29.288553 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:29.288480    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh" event={"ID":"68ff145f-d0fc-42af-8688-f25544f2809d","Type":"ContainerStarted","Data":"98af217304e99660b1479bbf97badf6f78214e138bb311f32bebd9005b29e3c6"}
Oct 29 21:51:31.704920 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:31.704721    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:31.838006 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:31.837931    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:31.958755 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:31.958608    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:32.119670 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:32.119633    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:32.246917 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:32.246841    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:33.326471 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:33.326427    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-xcv5v" event={"ID":"e3059381-b63e-4d76-bf2d-ade2250ff36a","Type":"ContainerStarted","Data":"9a795b657c7806b0e129c21f06642f6d0cb61573d13f3c98a718cb5831677630"}
Oct 29 21:51:33.327422 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:33.327396    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-xcv5v"
Oct 29 21:51:33.371527 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:33.371463    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-xcv5v" podStartSLOduration=1.7387238219999999 podStartE2EDuration="5.3714442s" totalImagesPullingTime="3.632720378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:51:33.370093981 +0000 UTC m=+309.527219479" watchObservedRunningTime="2026-10-29 21:51:33.3714442 +0000 UTC m=+309.528569699"
Oct 29 21:51:34.335906 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:34.335366    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh" event={"ID":"68ff145f-d0fc-42af-8688-f25544f2809d","Type":"ContainerStarted","Data":"476588e925cc5f2f4fd0bbb3239cd9e1eecd46d224c616f1d921630810efc915"}
Oct 29 21:51:34.335906 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:34.335539    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"
Oct 29 21:51:34.340190 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:34.340159    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-f5cm2" event={"ID":"20102b85-6195-42e4-94ea-873d5af3ad35","Type":"ContainerStarted","Data":"5cf558c916f6fabe9de1fb1303e63519a5a624b424152bd82ad46df972e356a5"}
Oct 29 21:51:34.340328 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:34.340212    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-f5cm2"
Oct 29 21:51:34.379508 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:34.379450    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh" podStartSLOduration=2.091431622 podStartE2EDuration="6.379431933s" totalImagesPullingTime="4.288000311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:51:34.378604905 +0000 UTC m=+310.535730432" watchObservedRunningTime="2026-10-29 21:51:34.379431933 +0000 UTC m=+310.536557432"
Oct 29 21:51:34.422681 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:34.422627    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-f5cm2" podStartSLOduration=2.099284992 podStartE2EDuration="6.422608584s" totalImagesPullingTime="4.323323592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:51:34.421017772 +0000 UTC m=+310.578143273" watchObservedRunningTime="2026-10-29 21:51:34.422608584 +0000 UTC m=+310.579734081"
Oct 29 21:51:35.267635 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:35.267595    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7"
Oct 29 21:51:35.355376 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:35.355338    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-8jrxh" event={"ID":"b10e7c81-07e3-4c64-8433-ee31a9a26fbb","Type":"ContainerStarted","Data":"3609780eb10f2dbf22d9349a6169ca0e9bf5fe5daa938585d38ad3dfa4fec19a"}
Oct 29 21:51:35.356370 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:35.356310    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-8jrxh"
Oct 29 21:51:35.400425 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:35.400366    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-8jrxh" podStartSLOduration=2.17189713 podStartE2EDuration="7.400345587s" totalImagesPullingTime="5.228448457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:51:35.399086423 +0000 UTC m=+311.556211922" watchObservedRunningTime="2026-10-29 21:51:35.400345587 +0000 UTC m=+311.557471087"
Oct 29 21:51:36.266995 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:36.266960    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f6b55dfc4-hv5c5"
Oct 29 21:51:36.343442 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:36.343406    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7cb76cfbb6-fkcm7"]
Oct 29 21:51:36.343629 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:36.343601    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7" podUID="8d517c7c-3570-4570-ad81-26ad57874d6d" containerName="manager" containerID="cri-o://a07dfabb34c5cdf70cd2820f464749301d2d89890e9830deaf13f7ccd9ebdf54" gracePeriod=30
Oct 29 21:51:36.647138 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:36.647107    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7"
Oct 29 21:51:36.692397 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:36.691306    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d517c7c-3570-4570-ad81-26ad57874d6d-kube-api-access-hq64t\" (UniqueName: \"kubernetes.io/projected/8d517c7c-3570-4570-ad81-26ad57874d6d-kube-api-access-hq64t\") pod \"8d517c7c-3570-4570-ad81-26ad57874d6d\" (UID: \"8d517c7c-3570-4570-ad81-26ad57874d6d\") "
Oct 29 21:51:36.692397 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:36.691378    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d517c7c-3570-4570-ad81-26ad57874d6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d517c7c-3570-4570-ad81-26ad57874d6d-metrics-certs\") pod \"8d517c7c-3570-4570-ad81-26ad57874d6d\" (UID: \"8d517c7c-3570-4570-ad81-26ad57874d6d\") "
Oct 29 21:51:36.692397 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:36.691452    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d517c7c-3570-4570-ad81-26ad57874d6d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d517c7c-3570-4570-ad81-26ad57874d6d-webhook-cert\") pod \"8d517c7c-3570-4570-ad81-26ad57874d6d\" (UID: \"8d517c7c-3570-4570-ad81-26ad57874d6d\") "
Oct 29 21:51:36.705735 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:36.705677    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d517c7c-3570-4570-ad81-26ad57874d6d-metrics-certs" pod "8d517c7c-3570-4570-ad81-26ad57874d6d" (UID: "8d517c7c-3570-4570-ad81-26ad57874d6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:51:36.708283 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:36.708238    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d517c7c-3570-4570-ad81-26ad57874d6d-webhook-cert" pod "8d517c7c-3570-4570-ad81-26ad57874d6d" (UID: "8d517c7c-3570-4570-ad81-26ad57874d6d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:51:36.708651 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:36.708601    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d517c7c-3570-4570-ad81-26ad57874d6d-kube-api-access-hq64t" pod "8d517c7c-3570-4570-ad81-26ad57874d6d" (UID: "8d517c7c-3570-4570-ad81-26ad57874d6d"). InnerVolumeSpecName "kube-api-access-hq64t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:51:36.792673 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:36.792591    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq64t\" (UniqueName: \"kubernetes.io/projected/8d517c7c-3570-4570-ad81-26ad57874d6d-kube-api-access-hq64t\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:51:36.792673 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:36.792627    4550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d517c7c-3570-4570-ad81-26ad57874d6d-metrics-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:51:36.792673 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:36.792640    4550 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d517c7c-3570-4570-ad81-26ad57874d6d-webhook-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:51:37.368562 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:37.368519    4550 generic.go:356] "Generic (PLEG): container finished" podID="8d517c7c-3570-4570-ad81-26ad57874d6d" containerID="a07dfabb34c5cdf70cd2820f464749301d2d89890e9830deaf13f7ccd9ebdf54" exitCode=0
Oct 29 21:51:37.368741 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:37.368575    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7"
Oct 29 21:51:37.368741 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:37.368607    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7" event={"ID":"8d517c7c-3570-4570-ad81-26ad57874d6d","Type":"ContainerDied","Data":"a07dfabb34c5cdf70cd2820f464749301d2d89890e9830deaf13f7ccd9ebdf54"}
Oct 29 21:51:37.368741 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:37.368683    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cb76cfbb6-fkcm7" event={"ID":"8d517c7c-3570-4570-ad81-26ad57874d6d","Type":"ContainerDied","Data":"c63b6a76a12434577ebcbf15b90b10e5e98fa29cf1ac6c816cf568565f2800c3"}
Oct 29 21:51:37.368741 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:37.368714    4550 scope.go:122] "RemoveContainer" containerID="a07dfabb34c5cdf70cd2820f464749301d2d89890e9830deaf13f7ccd9ebdf54"
Oct 29 21:51:37.380135 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:37.380114    4550 scope.go:122] "RemoveContainer" containerID="a07dfabb34c5cdf70cd2820f464749301d2d89890e9830deaf13f7ccd9ebdf54"
Oct 29 21:51:37.380524 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:51:37.380492    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07dfabb34c5cdf70cd2820f464749301d2d89890e9830deaf13f7ccd9ebdf54\": container with ID starting with a07dfabb34c5cdf70cd2820f464749301d2d89890e9830deaf13f7ccd9ebdf54 not found: ID does not exist" containerID="a07dfabb34c5cdf70cd2820f464749301d2d89890e9830deaf13f7ccd9ebdf54"
Oct 29 21:51:37.380638 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:37.380534    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07dfabb34c5cdf70cd2820f464749301d2d89890e9830deaf13f7ccd9ebdf54"} err="failed to get container status \"a07dfabb34c5cdf70cd2820f464749301d2d89890e9830deaf13f7ccd9ebdf54\": rpc error: code = NotFound desc = could not find container \"a07dfabb34c5cdf70cd2820f464749301d2d89890e9830deaf13f7ccd9ebdf54\": container with ID starting with a07dfabb34c5cdf70cd2820f464749301d2d89890e9830deaf13f7ccd9ebdf54 not found: ID does not exist"
Oct 29 21:51:37.419984 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:37.419955    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7cb76cfbb6-fkcm7"]
Oct 29 21:51:37.432245 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:37.432225    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7cb76cfbb6-fkcm7"]
Oct 29 21:51:37.993527 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:37.993483    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d517c7c-3570-4570-ad81-26ad57874d6d" path="/var/lib/kubelet/pods/8d517c7c-3570-4570-ad81-26ad57874d6d/volumes"
Oct 29 21:51:41.391815 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:41.391780    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2" event={"ID":"a11c33aa-7a9a-422f-8697-a36ef4bc30ba","Type":"ContainerStarted","Data":"2ca426a570f98fe68a33f57389e43d2dc308569edb3dd686b4e63a77fef07b83"}
Oct 29 21:51:41.392258 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:41.391884    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:41.436200 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:41.436158    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2" podStartSLOduration=1.435778838 podStartE2EDuration="14.43613272s" totalImagesPullingTime="13.000353882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:51:41.434886487 +0000 UTC m=+317.592011987" watchObservedRunningTime="2026-10-29 21:51:41.43613272 +0000 UTC m=+317.593258217"
Oct 29 21:51:41.497403 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:51:41.497361    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:51:45.359972 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:45.359943    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-xcv5v"
Oct 29 21:51:47.370854 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:47.370821    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-8jrxh"
Oct 29 21:51:52.401611 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:52.401580    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r8ft2"
Oct 29 21:51:53.181255 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:53.181211    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:55.020443 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:55.020402    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:51:55.175495 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:51:55.175464    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:52:05.046679 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.046643    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:52:05.168256 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.168220    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:52:05.357037 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.357007    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"
Oct 29 21:52:05.361034 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.361009    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-f5cm2"
Oct 29 21:52:05.642514 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.642442    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"]
Oct 29 21:52:05.642839 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.642826    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8d517c7c-3570-4570-ad81-26ad57874d6d" containerName="manager"
Oct 29 21:52:05.642883 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.642841    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d517c7c-3570-4570-ad81-26ad57874d6d" containerName="manager"
Oct 29 21:52:05.642917 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.642903    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d517c7c-3570-4570-ad81-26ad57874d6d" containerName="manager"
Oct 29 21:52:05.647278 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.647259    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:05.664027 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.664009    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 29 21:52:05.666998 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.666979    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2qkjk\""
Oct 29 21:52:05.667117 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.667007    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 21:52:05.677792 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.677769    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"]
Oct 29 21:52:05.741725 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.741700    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/195d1fda-1571-42d4-9c8e-7ecfebdaeade-metrics-certs\") pod \"maas-api-5569c899b4-gr72l\" (UID: \"195d1fda-1571-42d4-9c8e-7ecfebdaeade\") " pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:05.741848 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.741746    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195d1fda-1571-42d4-9c8e-7ecfebdaeade-openshift-service-ca\") pod \"maas-api-5569c899b4-gr72l\" (UID: \"195d1fda-1571-42d4-9c8e-7ecfebdaeade\") " pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:05.741896 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.741882    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195d1fda-1571-42d4-9c8e-7ecfebdaeade-maas-api-tls\") pod \"maas-api-5569c899b4-gr72l\" (UID: \"195d1fda-1571-42d4-9c8e-7ecfebdaeade\") " pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:05.741932 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.741922    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c49v\" (UniqueName: \"kubernetes.io/projected/195d1fda-1571-42d4-9c8e-7ecfebdaeade-kube-api-access-4c49v\") pod \"maas-api-5569c899b4-gr72l\" (UID: \"195d1fda-1571-42d4-9c8e-7ecfebdaeade\") " pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:05.831617 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.831589    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d675dcc65-fj2qz"]
Oct 29 21:52:05.834927 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.834910    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d675dcc65-fj2qz"
Oct 29 21:52:05.843048 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.843024    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195d1fda-1571-42d4-9c8e-7ecfebdaeade-maas-api-tls\") pod \"maas-api-5569c899b4-gr72l\" (UID: \"195d1fda-1571-42d4-9c8e-7ecfebdaeade\") " pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:05.843132 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.843070    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c49v\" (UniqueName: \"kubernetes.io/projected/195d1fda-1571-42d4-9c8e-7ecfebdaeade-kube-api-access-4c49v\") pod \"maas-api-5569c899b4-gr72l\" (UID: \"195d1fda-1571-42d4-9c8e-7ecfebdaeade\") " pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:05.843132 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.843099    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/195d1fda-1571-42d4-9c8e-7ecfebdaeade-metrics-certs\") pod \"maas-api-5569c899b4-gr72l\" (UID: \"195d1fda-1571-42d4-9c8e-7ecfebdaeade\") " pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:05.843132 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.843123    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195d1fda-1571-42d4-9c8e-7ecfebdaeade-openshift-service-ca\") pod \"maas-api-5569c899b4-gr72l\" (UID: \"195d1fda-1571-42d4-9c8e-7ecfebdaeade\") " pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:05.843815 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.843796    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195d1fda-1571-42d4-9c8e-7ecfebdaeade-openshift-service-ca\") pod \"maas-api-5569c899b4-gr72l\" (UID: \"195d1fda-1571-42d4-9c8e-7ecfebdaeade\") " pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:05.845491 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.845466    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/195d1fda-1571-42d4-9c8e-7ecfebdaeade-metrics-certs\") pod \"maas-api-5569c899b4-gr72l\" (UID: \"195d1fda-1571-42d4-9c8e-7ecfebdaeade\") " pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:05.845617 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.845597    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195d1fda-1571-42d4-9c8e-7ecfebdaeade-maas-api-tls\") pod \"maas-api-5569c899b4-gr72l\" (UID: \"195d1fda-1571-42d4-9c8e-7ecfebdaeade\") " pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:05.856330 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.856305    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d675dcc65-fj2qz"]
Oct 29 21:52:05.864960 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.864942    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-h7r8w\""
Oct 29 21:52:05.865053 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.865023    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 29 21:52:05.866988 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.866968    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c49v\" (UniqueName: \"kubernetes.io/projected/195d1fda-1571-42d4-9c8e-7ecfebdaeade-kube-api-access-4c49v\") pod \"maas-api-5569c899b4-gr72l\" (UID: \"195d1fda-1571-42d4-9c8e-7ecfebdaeade\") " pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:05.944066 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.944005    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fcd91abe-004a-48b9-9fea-29a964ef2eec-tls-cert\") pod \"authorino-d675dcc65-fj2qz\" (UID: \"fcd91abe-004a-48b9-9fea-29a964ef2eec\") " pod="kuadrant-system/authorino-d675dcc65-fj2qz"
Oct 29 21:52:05.944066 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.944036    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf7fz\" (UniqueName: \"kubernetes.io/projected/fcd91abe-004a-48b9-9fea-29a964ef2eec-kube-api-access-jf7fz\") pod \"authorino-d675dcc65-fj2qz\" (UID: \"fcd91abe-004a-48b9-9fea-29a964ef2eec\") " pod="kuadrant-system/authorino-d675dcc65-fj2qz"
Oct 29 21:52:05.960501 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:05.960480    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:06.045349 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:06.045317    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fcd91abe-004a-48b9-9fea-29a964ef2eec-tls-cert\") pod \"authorino-d675dcc65-fj2qz\" (UID: \"fcd91abe-004a-48b9-9fea-29a964ef2eec\") " pod="kuadrant-system/authorino-d675dcc65-fj2qz"
Oct 29 21:52:06.045501 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:06.045353    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf7fz\" (UniqueName: \"kubernetes.io/projected/fcd91abe-004a-48b9-9fea-29a964ef2eec-kube-api-access-jf7fz\") pod \"authorino-d675dcc65-fj2qz\" (UID: \"fcd91abe-004a-48b9-9fea-29a964ef2eec\") " pod="kuadrant-system/authorino-d675dcc65-fj2qz"
Oct 29 21:52:06.047671 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:06.047647    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fcd91abe-004a-48b9-9fea-29a964ef2eec-tls-cert\") pod \"authorino-d675dcc65-fj2qz\" (UID: \"fcd91abe-004a-48b9-9fea-29a964ef2eec\") " pod="kuadrant-system/authorino-d675dcc65-fj2qz"
Oct 29 21:52:06.069164 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:06.069130    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf7fz\" (UniqueName: \"kubernetes.io/projected/fcd91abe-004a-48b9-9fea-29a964ef2eec-kube-api-access-jf7fz\") pod \"authorino-d675dcc65-fj2qz\" (UID: \"fcd91abe-004a-48b9-9fea-29a964ef2eec\") " pod="kuadrant-system/authorino-d675dcc65-fj2qz"
Oct 29 21:52:06.146311 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:06.146276    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d675dcc65-fj2qz"
Oct 29 21:52:06.264194 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:06.264171    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"]
Oct 29 21:52:06.265937 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:52:06.265910    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195d1fda_1571_42d4_9c8e_7ecfebdaeade.slice/crio-cef4f3c52da396a5e71de35a930a44454f0cf992ef891f4d123b8bb57d4caf33 WatchSource:0}: Error finding container cef4f3c52da396a5e71de35a930a44454f0cf992ef891f4d123b8bb57d4caf33: Status 404 returned error can't find the container with id cef4f3c52da396a5e71de35a930a44454f0cf992ef891f4d123b8bb57d4caf33
Oct 29 21:52:06.442709 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:06.442685    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d675dcc65-fj2qz"]
Oct 29 21:52:06.445137 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:52:06.445061    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcd91abe_004a_48b9_9fea_29a964ef2eec.slice/crio-5e2b9deb7251e8e53ca6e14bb326208f6b6b83bc99a5ce788189f1360dae2dd6 WatchSource:0}: Error finding container 5e2b9deb7251e8e53ca6e14bb326208f6b6b83bc99a5ce788189f1360dae2dd6: Status 404 returned error can't find the container with id 5e2b9deb7251e8e53ca6e14bb326208f6b6b83bc99a5ce788189f1360dae2dd6
Oct 29 21:52:06.501295 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:06.501264    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d675dcc65-fj2qz" event={"ID":"fcd91abe-004a-48b9-9fea-29a964ef2eec","Type":"ContainerStarted","Data":"5e2b9deb7251e8e53ca6e14bb326208f6b6b83bc99a5ce788189f1360dae2dd6"}
Oct 29 21:52:06.502439 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:06.502416    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l" event={"ID":"195d1fda-1571-42d4-9c8e-7ecfebdaeade","Type":"ContainerStarted","Data":"cef4f3c52da396a5e71de35a930a44454f0cf992ef891f4d123b8bb57d4caf33"}
Oct 29 21:52:06.684913 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:06.684876    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d675dcc65-fj2qz"]
Oct 29 21:52:07.509214 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:07.509167    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d675dcc65-fj2qz" event={"ID":"fcd91abe-004a-48b9-9fea-29a964ef2eec","Type":"ContainerStarted","Data":"867c8f3adc2ff2c223adc627bda2507332df875b642632773ead6060626f0b21"}
Oct 29 21:52:07.509674 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:07.509265    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-d675dcc65-fj2qz" podUID="fcd91abe-004a-48b9-9fea-29a964ef2eec" containerName="authorino" containerID="cri-o://867c8f3adc2ff2c223adc627bda2507332df875b642632773ead6060626f0b21" gracePeriod=30
Oct 29 21:52:07.553716 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:07.553668    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-d675dcc65-fj2qz" podStartSLOduration=2.027051001 podStartE2EDuration="2.55365328s" totalImagesPullingTime="526.602279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:07.551387423 +0000 UTC m=+343.708512959" watchObservedRunningTime="2026-10-29 21:52:07.55365328 +0000 UTC m=+343.710778778"
Oct 29 21:52:07.822651 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:07.822621    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d675dcc65-fj2qz"
Oct 29 21:52:07.966307 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:07.966272    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd91abe-004a-48b9-9fea-29a964ef2eec-kube-api-access-jf7fz\" (UniqueName: \"kubernetes.io/projected/fcd91abe-004a-48b9-9fea-29a964ef2eec-kube-api-access-jf7fz\") pod \"fcd91abe-004a-48b9-9fea-29a964ef2eec\" (UID: \"fcd91abe-004a-48b9-9fea-29a964ef2eec\") "
Oct 29 21:52:07.966485 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:07.966348    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcd91abe-004a-48b9-9fea-29a964ef2eec-tls-cert\" (UniqueName: \"kubernetes.io/secret/fcd91abe-004a-48b9-9fea-29a964ef2eec-tls-cert\") pod \"fcd91abe-004a-48b9-9fea-29a964ef2eec\" (UID: \"fcd91abe-004a-48b9-9fea-29a964ef2eec\") "
Oct 29 21:52:07.968287 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:07.968261    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd91abe-004a-48b9-9fea-29a964ef2eec-kube-api-access-jf7fz" pod "fcd91abe-004a-48b9-9fea-29a964ef2eec" (UID: "fcd91abe-004a-48b9-9fea-29a964ef2eec"). InnerVolumeSpecName "kube-api-access-jf7fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:52:07.976097 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:07.976070    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcd91abe-004a-48b9-9fea-29a964ef2eec-tls-cert" pod "fcd91abe-004a-48b9-9fea-29a964ef2eec" (UID: "fcd91abe-004a-48b9-9fea-29a964ef2eec"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:52:08.067131 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:08.067076    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf7fz\" (UniqueName: \"kubernetes.io/projected/fcd91abe-004a-48b9-9fea-29a964ef2eec-kube-api-access-jf7fz\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:52:08.067131 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:08.067101    4550 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fcd91abe-004a-48b9-9fea-29a964ef2eec-tls-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:52:08.516401 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:08.516368    4550 generic.go:356] "Generic (PLEG): container finished" podID="fcd91abe-004a-48b9-9fea-29a964ef2eec" containerID="867c8f3adc2ff2c223adc627bda2507332df875b642632773ead6060626f0b21" exitCode=0
Oct 29 21:52:08.516882 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:08.516410    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d675dcc65-fj2qz"
Oct 29 21:52:08.516882 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:08.516453    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d675dcc65-fj2qz" event={"ID":"fcd91abe-004a-48b9-9fea-29a964ef2eec","Type":"ContainerDied","Data":"867c8f3adc2ff2c223adc627bda2507332df875b642632773ead6060626f0b21"}
Oct 29 21:52:08.516882 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:08.516498    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d675dcc65-fj2qz" event={"ID":"fcd91abe-004a-48b9-9fea-29a964ef2eec","Type":"ContainerDied","Data":"5e2b9deb7251e8e53ca6e14bb326208f6b6b83bc99a5ce788189f1360dae2dd6"}
Oct 29 21:52:08.516882 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:08.516520    4550 scope.go:122] "RemoveContainer" containerID="867c8f3adc2ff2c223adc627bda2507332df875b642632773ead6060626f0b21"
Oct 29 21:52:08.518812 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:08.518784    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l" event={"ID":"195d1fda-1571-42d4-9c8e-7ecfebdaeade","Type":"ContainerStarted","Data":"af0e63a40d10a36f7fd3170176ec7a93f8812f3ec64c92735ea0faaa30f93d8a"}
Oct 29 21:52:08.518918 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:08.518885    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:08.524951 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:08.524932    4550 scope.go:122] "RemoveContainer" containerID="867c8f3adc2ff2c223adc627bda2507332df875b642632773ead6060626f0b21"
Oct 29 21:52:08.525291 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:52:08.525268    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"867c8f3adc2ff2c223adc627bda2507332df875b642632773ead6060626f0b21\": container with ID starting with 867c8f3adc2ff2c223adc627bda2507332df875b642632773ead6060626f0b21 not found: ID does not exist" containerID="867c8f3adc2ff2c223adc627bda2507332df875b642632773ead6060626f0b21"
Oct 29 21:52:08.525373 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:08.525299    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"867c8f3adc2ff2c223adc627bda2507332df875b642632773ead6060626f0b21"} err="failed to get container status \"867c8f3adc2ff2c223adc627bda2507332df875b642632773ead6060626f0b21\": rpc error: code = NotFound desc = could not find container \"867c8f3adc2ff2c223adc627bda2507332df875b642632773ead6060626f0b21\": container with ID starting with 867c8f3adc2ff2c223adc627bda2507332df875b642632773ead6060626f0b21 not found: ID does not exist"
Oct 29 21:52:08.572465 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:08.572397    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d675dcc65-fj2qz"]
Oct 29 21:52:08.584416 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:08.584392    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d675dcc65-fj2qz"]
Oct 29 21:52:08.628913 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:08.628864    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l" podStartSLOduration=1.6491057740000001 podStartE2EDuration="3.628850539s" totalImagesPullingTime="1.979744765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:08.627734644 +0000 UTC m=+344.784860162" watchObservedRunningTime="2026-10-29 21:52:08.628850539 +0000 UTC m=+344.785976038"
Oct 29 21:52:09.919737 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:09.919707    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:52:09.987154 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:09.987119    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd91abe-004a-48b9-9fea-29a964ef2eec" path="/var/lib/kubelet/pods/fcd91abe-004a-48b9-9fea-29a964ef2eec/volumes"
Oct 29 21:52:11.489888 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:52:11.489852    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:52:14.530664 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:14.530629    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5569c899b4-gr72l"
Oct 29 21:52:18.679564 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:18.679527    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:52:18.803704 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:18.803673    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:52:29.662652 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:29.662617    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-q94rt"]
Oct 29 21:52:29.663021 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:29.662996    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fcd91abe-004a-48b9-9fea-29a964ef2eec" containerName="authorino"
Oct 29 21:52:29.663021 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:29.663009    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd91abe-004a-48b9-9fea-29a964ef2eec" containerName="authorino"
Oct 29 21:52:29.663097 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:29.663088    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd91abe-004a-48b9-9fea-29a964ef2eec" containerName="authorino"
Oct 29 21:52:29.690868 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:29.690843    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-q94rt"]
Oct 29 21:52:29.691011 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:29.690944    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q94rt"
Oct 29 21:52:29.761017 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:29.760991    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd3a6ac7-2936-45af-9951-a1002cbeab78-cert\") pod \"llmisvc-controller-manager-6466c554d8-q94rt\" (UID: \"cd3a6ac7-2936-45af-9951-a1002cbeab78\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q94rt"
Oct 29 21:52:29.761185 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:29.761051    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg7ps\" (UniqueName: \"kubernetes.io/projected/cd3a6ac7-2936-45af-9951-a1002cbeab78-kube-api-access-zg7ps\") pod \"llmisvc-controller-manager-6466c554d8-q94rt\" (UID: \"cd3a6ac7-2936-45af-9951-a1002cbeab78\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q94rt"
Oct 29 21:52:29.862512 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:29.862478    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd3a6ac7-2936-45af-9951-a1002cbeab78-cert\") pod \"llmisvc-controller-manager-6466c554d8-q94rt\" (UID: \"cd3a6ac7-2936-45af-9951-a1002cbeab78\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q94rt"
Oct 29 21:52:29.862646 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:29.862533    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg7ps\" (UniqueName: \"kubernetes.io/projected/cd3a6ac7-2936-45af-9951-a1002cbeab78-kube-api-access-zg7ps\") pod \"llmisvc-controller-manager-6466c554d8-q94rt\" (UID: \"cd3a6ac7-2936-45af-9951-a1002cbeab78\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q94rt"
Oct 29 21:52:29.864813 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:29.864788    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd3a6ac7-2936-45af-9951-a1002cbeab78-cert\") pod \"llmisvc-controller-manager-6466c554d8-q94rt\" (UID: \"cd3a6ac7-2936-45af-9951-a1002cbeab78\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q94rt"
Oct 29 21:52:29.886476 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:29.886451    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg7ps\" (UniqueName: \"kubernetes.io/projected/cd3a6ac7-2936-45af-9951-a1002cbeab78-kube-api-access-zg7ps\") pod \"llmisvc-controller-manager-6466c554d8-q94rt\" (UID: \"cd3a6ac7-2936-45af-9951-a1002cbeab78\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q94rt"
Oct 29 21:52:30.002298 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:30.002239    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q94rt"
Oct 29 21:52:30.300854 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:30.300826    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-q94rt"]
Oct 29 21:52:30.302809 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:52:30.302778    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3a6ac7_2936_45af_9951_a1002cbeab78.slice/crio-a701c2027bd088c0962b8808414242241b32d982f38c695729ce4c3d140e6cc8 WatchSource:0}: Error finding container a701c2027bd088c0962b8808414242241b32d982f38c695729ce4c3d140e6cc8: Status 404 returned error can't find the container with id a701c2027bd088c0962b8808414242241b32d982f38c695729ce4c3d140e6cc8
Oct 29 21:52:30.617270 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:30.617236    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q94rt" event={"ID":"cd3a6ac7-2936-45af-9951-a1002cbeab78","Type":"ContainerStarted","Data":"a701c2027bd088c0962b8808414242241b32d982f38c695729ce4c3d140e6cc8"}
Oct 29 21:52:31.622892 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:31.622855    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q94rt" event={"ID":"cd3a6ac7-2936-45af-9951-a1002cbeab78","Type":"ContainerStarted","Data":"cbbb33e1f0a085560e9a290c5af945388e74e2b410d87899e28ed19e584572ce"}
Oct 29 21:52:31.623291 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:31.622970    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q94rt"
Oct 29 21:52:31.672830 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:31.672784    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q94rt" podStartSLOduration=2.316051455 podStartE2EDuration="2.672770333s" totalImagesPullingTime="356.718878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:31.670762191 +0000 UTC m=+367.827887689" watchObservedRunningTime="2026-10-29 21:52:31.672770333 +0000 UTC m=+367.829895830"
Oct 29 21:52:41.489507 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:52:41.489464    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:52:42.522802 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.522766    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"]
Oct 29 21:52:42.526950 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.526933    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.541835 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.541811    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 29 21:52:42.556096 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.556066    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"]
Oct 29 21:52:42.627660 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.627629    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b4c9t\""
Oct 29 21:52:42.669386 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.669355    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c05fddc0-c933-46fa-9dd4-e7881488c8a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.669533 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.669389    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05fddc0-c933-46fa-9dd4-e7881488c8a5-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.669533 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.669411    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05fddc0-c933-46fa-9dd4-e7881488c8a5-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.669533 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.669489    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5rtn\" (UniqueName: \"kubernetes.io/projected/c05fddc0-c933-46fa-9dd4-e7881488c8a5-kube-api-access-m5rtn\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.669710 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.669614    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c05fddc0-c933-46fa-9dd4-e7881488c8a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.669710 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.669664    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c05fddc0-c933-46fa-9dd4-e7881488c8a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.669806 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.669738    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c05fddc0-c933-46fa-9dd4-e7881488c8a5-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.669806 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.669791    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c05fddc0-c933-46fa-9dd4-e7881488c8a5-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.669903 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.669825    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c05fddc0-c933-46fa-9dd4-e7881488c8a5-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.771255 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.771226    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c05fddc0-c933-46fa-9dd4-e7881488c8a5-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.771392 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.771266    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c05fddc0-c933-46fa-9dd4-e7881488c8a5-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.771392 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.771325    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c05fddc0-c933-46fa-9dd4-e7881488c8a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.771392 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.771353    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05fddc0-c933-46fa-9dd4-e7881488c8a5-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.771392 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.771382    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05fddc0-c933-46fa-9dd4-e7881488c8a5-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.771606 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.771416    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5rtn\" (UniqueName: \"kubernetes.io/projected/c05fddc0-c933-46fa-9dd4-e7881488c8a5-kube-api-access-m5rtn\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.771606 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.771457    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c05fddc0-c933-46fa-9dd4-e7881488c8a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.771606 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.771500    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c05fddc0-c933-46fa-9dd4-e7881488c8a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.771606 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.771533    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c05fddc0-c933-46fa-9dd4-e7881488c8a5-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.771788 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.771662    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c05fddc0-c933-46fa-9dd4-e7881488c8a5-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.771788 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.771741    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05fddc0-c933-46fa-9dd4-e7881488c8a5-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.771932 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.771910    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05fddc0-c933-46fa-9dd4-e7881488c8a5-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.771985 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.771961    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c05fddc0-c933-46fa-9dd4-e7881488c8a5-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.772183 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.772162    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c05fddc0-c933-46fa-9dd4-e7881488c8a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.773727 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.773680    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c05fddc0-c933-46fa-9dd4-e7881488c8a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.773937 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.773918    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c05fddc0-c933-46fa-9dd4-e7881488c8a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.794712 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.794686    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c05fddc0-c933-46fa-9dd4-e7881488c8a5-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.794911 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.794892    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5rtn\" (UniqueName: \"kubernetes.io/projected/c05fddc0-c933-46fa-9dd4-e7881488c8a5-kube-api-access-m5rtn\") pod \"maas-default-gateway-openshift-default-769df7846c-b4jg2\" (UID: \"c05fddc0-c933-46fa-9dd4-e7881488c8a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:42.839856 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:42.839835    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:43.162764 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:43.162738    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"]
Oct 29 21:52:43.164794 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:52:43.164766    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05fddc0_c933_46fa_9dd4_e7881488c8a5.slice/crio-b029391f4a799fd4d5aa235ee4c19f49610609a3ca462e4e95a6199486154421 WatchSource:0}: Error finding container b029391f4a799fd4d5aa235ee4c19f49610609a3ca462e4e95a6199486154421: Status 404 returned error can't find the container with id b029391f4a799fd4d5aa235ee4c19f49610609a3ca462e4e95a6199486154421
Oct 29 21:52:43.674680 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:43.674643    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2" event={"ID":"c05fddc0-c933-46fa-9dd4-e7881488c8a5","Type":"ContainerStarted","Data":"b029391f4a799fd4d5aa235ee4c19f49610609a3ca462e4e95a6199486154421"}
Oct 29 21:52:45.847517 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:45.847484    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:52:45.847811 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:45.847552    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:52:45.847811 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:45.847581    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:52:46.691950 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:46.691910    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2" event={"ID":"c05fddc0-c933-46fa-9dd4-e7881488c8a5","Type":"ContainerStarted","Data":"ab7a792d51ee5b7ab67617188923a489c1281b45a6fd65a74c9b0d8ba77aa92b"}
Oct 29 21:52:46.750997 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:46.750946    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2" podStartSLOduration=2.070935484 podStartE2EDuration="4.750933789s" totalImagesPullingTime="2.679998305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:46.750606746 +0000 UTC m=+382.907732255" watchObservedRunningTime="2026-10-29 21:52:46.750933789 +0000 UTC m=+382.908059287"
Oct 29 21:52:46.840618 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:46.840585    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:46.842205 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:46.842178    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b4jg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 29 21:52:46.842346 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:46.842232    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2" podUID="c05fddc0-c933-46fa-9dd4-e7881488c8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 29 21:52:47.840648 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:47.840612    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b4jg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 29 21:52:47.841025 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:47.840670    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2" podUID="c05fddc0-c933-46fa-9dd4-e7881488c8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 29 21:52:48.840715 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:48.840681    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b4jg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 29 21:52:48.841134 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:48.840731    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2" podUID="c05fddc0-c933-46fa-9dd4-e7881488c8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 29 21:52:49.840898 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:49.840862    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b4jg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 29 21:52:49.841349 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:49.840929    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2" podUID="c05fddc0-c933-46fa-9dd4-e7881488c8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 29 21:52:50.840309 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:50.840269    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b4jg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 29 21:52:50.840515 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:50.840333    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2" podUID="c05fddc0-c933-46fa-9dd4-e7881488c8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 29 21:52:51.840648 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:51.840611    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b4jg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 29 21:52:51.841082 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:51.840687    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2" podUID="c05fddc0-c933-46fa-9dd4-e7881488c8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 29 21:52:53.088402 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:53.088369    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:53.090067 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:53.090034    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:52:53.091033 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:52:53.091000    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b4jg2"
Oct 29 21:53:02.628547 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:02.628515    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q94rt"
Oct 29 21:53:02.705615 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:02.705578    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"]
Oct 29 21:53:02.705831 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:02.705783    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh" podUID="68ff145f-d0fc-42af-8688-f25544f2809d" containerName="manager" containerID="cri-o://476588e925cc5f2f4fd0bbb3239cd9e1eecd46d224c616f1d921630810efc915" gracePeriod=30
Oct 29 21:53:02.951228 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:02.951200    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"
Oct 29 21:53:03.048033 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.048003    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ff145f-d0fc-42af-8688-f25544f2809d-kube-api-access-dj6xp\" (UniqueName: \"kubernetes.io/projected/68ff145f-d0fc-42af-8688-f25544f2809d-kube-api-access-dj6xp\") pod \"68ff145f-d0fc-42af-8688-f25544f2809d\" (UID: \"68ff145f-d0fc-42af-8688-f25544f2809d\") "
Oct 29 21:53:03.048199 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.048095    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68ff145f-d0fc-42af-8688-f25544f2809d-cert\" (UniqueName: \"kubernetes.io/secret/68ff145f-d0fc-42af-8688-f25544f2809d-cert\") pod \"68ff145f-d0fc-42af-8688-f25544f2809d\" (UID: \"68ff145f-d0fc-42af-8688-f25544f2809d\") "
Oct 29 21:53:03.050155 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.050119    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ff145f-d0fc-42af-8688-f25544f2809d-cert" pod "68ff145f-d0fc-42af-8688-f25544f2809d" (UID: "68ff145f-d0fc-42af-8688-f25544f2809d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:53:03.050276 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.050255    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ff145f-d0fc-42af-8688-f25544f2809d-kube-api-access-dj6xp" pod "68ff145f-d0fc-42af-8688-f25544f2809d" (UID: "68ff145f-d0fc-42af-8688-f25544f2809d"). InnerVolumeSpecName "kube-api-access-dj6xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:53:03.149344 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.149308    4550 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68ff145f-d0fc-42af-8688-f25544f2809d-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:53:03.149344 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.149333    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj6xp\" (UniqueName: \"kubernetes.io/projected/68ff145f-d0fc-42af-8688-f25544f2809d-kube-api-access-dj6xp\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:53:03.774898 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.774868    4550 generic.go:356] "Generic (PLEG): container finished" podID="68ff145f-d0fc-42af-8688-f25544f2809d" containerID="476588e925cc5f2f4fd0bbb3239cd9e1eecd46d224c616f1d921630810efc915" exitCode=0
Oct 29 21:53:03.774898 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.774905    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh" event={"ID":"68ff145f-d0fc-42af-8688-f25544f2809d","Type":"ContainerDied","Data":"476588e925cc5f2f4fd0bbb3239cd9e1eecd46d224c616f1d921630810efc915"}
Oct 29 21:53:03.774898 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.774914    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"
Oct 29 21:53:03.775481 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.774947    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh" event={"ID":"68ff145f-d0fc-42af-8688-f25544f2809d","Type":"ContainerDied","Data":"98af217304e99660b1479bbf97badf6f78214e138bb311f32bebd9005b29e3c6"}
Oct 29 21:53:03.775481 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.774970    4550 scope.go:122] "RemoveContainer" containerID="476588e925cc5f2f4fd0bbb3239cd9e1eecd46d224c616f1d921630810efc915"
Oct 29 21:53:03.787596 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.786276    4550 scope.go:122] "RemoveContainer" containerID="476588e925cc5f2f4fd0bbb3239cd9e1eecd46d224c616f1d921630810efc915"
Oct 29 21:53:03.788164 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:53:03.788116    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476588e925cc5f2f4fd0bbb3239cd9e1eecd46d224c616f1d921630810efc915\": container with ID starting with 476588e925cc5f2f4fd0bbb3239cd9e1eecd46d224c616f1d921630810efc915 not found: ID does not exist" containerID="476588e925cc5f2f4fd0bbb3239cd9e1eecd46d224c616f1d921630810efc915"
Oct 29 21:53:03.788266 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.788175    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476588e925cc5f2f4fd0bbb3239cd9e1eecd46d224c616f1d921630810efc915"} err="failed to get container status \"476588e925cc5f2f4fd0bbb3239cd9e1eecd46d224c616f1d921630810efc915\": rpc error: code = NotFound desc = could not find container \"476588e925cc5f2f4fd0bbb3239cd9e1eecd46d224c616f1d921630810efc915\": container with ID starting with 476588e925cc5f2f4fd0bbb3239cd9e1eecd46d224c616f1d921630810efc915 not found: ID does not exist"
Oct 29 21:53:03.826421 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.826393    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"]
Oct 29 21:53:03.836977 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.836952    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4xfjh"]
Oct 29 21:53:03.864037 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.863993    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:53:03.988765 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:03.988731    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ff145f-d0fc-42af-8688-f25544f2809d" path="/var/lib/kubelet/pods/68ff145f-d0fc-42af-8688-f25544f2809d/volumes"
Oct 29 21:53:04.079459 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:04.079433    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:53:08.228910 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:08.228828    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:53:08.350412 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:08.350366    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:53:10.020694 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:10.020646    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:53:10.201992 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:10.201955    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:53:11.489973 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:53:11.489930    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:53:27.670037 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.669999    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"]
Oct 29 21:53:27.670506 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.670483    4550 cpu_manager.go:414] "containerMap: removing container" podUID="68ff145f-d0fc-42af-8688-f25544f2809d" containerName="manager"
Oct 29 21:53:27.670506 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.670504    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ff145f-d0fc-42af-8688-f25544f2809d" containerName="manager"
Oct 29 21:53:27.670652 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.670564    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ff145f-d0fc-42af-8688-f25544f2809d" containerName="manager"
Oct 29 21:53:27.679838 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.679821    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.710342 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.710318    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"]
Oct 29 21:53:27.711580 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.711559    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"]
Oct 29 21:53:27.715540 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.715519    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.734249 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.734221    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"]
Oct 29 21:53:27.763169 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763126    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.763287 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763221    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.763287 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763247    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.763372 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763316    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.763372 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763342    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.763466 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763372    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3deae563-f3ea-41cd-bd22-36ca90375806-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.763466 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763422    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64bss\" (UniqueName: \"kubernetes.io/projected/3deae563-f3ea-41cd-bd22-36ca90375806-kube-api-access-64bss\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.763566 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763473    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.763566 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763493    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.763566 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763509    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb930ab-251e-4279-9963-f6d0d02b5b98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.763675 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763588    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3deae563-f3ea-41cd-bd22-36ca90375806-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.763675 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763606    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.763675 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763626    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.763675 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763642    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.763859 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763738    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.763859 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763779    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3deae563-f3ea-41cd-bd22-36ca90375806-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.763859 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763825    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.763859 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.763844    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szphr\" (UniqueName: \"kubernetes.io/projected/5cb930ab-251e-4279-9963-f6d0d02b5b98-kube-api-access-szphr\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.864676 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.864652    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.864835 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.864683    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.864835 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.864700    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.864835 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.864718    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.864835 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.864742    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3deae563-f3ea-41cd-bd22-36ca90375806-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.864835 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.864766    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64bss\" (UniqueName: \"kubernetes.io/projected/3deae563-f3ea-41cd-bd22-36ca90375806-kube-api-access-64bss\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.864835 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.864820    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.865171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.864846    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.865171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.864868    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb930ab-251e-4279-9963-f6d0d02b5b98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.865171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.864910    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3deae563-f3ea-41cd-bd22-36ca90375806-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.865171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.864939    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.865171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.864979    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.865171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.865005    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.865171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.865061    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.865171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.865079    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.865171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.865100    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3deae563-f3ea-41cd-bd22-36ca90375806-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.865171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.865117    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.865667 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.865202    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.865667 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.865256    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.865667 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.865288    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szphr\" (UniqueName: \"kubernetes.io/projected/5cb930ab-251e-4279-9963-f6d0d02b5b98-kube-api-access-szphr\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.865667 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.865330    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.865667 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.865375    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.865667 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.865423    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.865982 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.865846    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb930ab-251e-4279-9963-f6d0d02b5b98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.865982 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.865954    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.866111 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.866080    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3deae563-f3ea-41cd-bd22-36ca90375806-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.866188 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.866131    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.866253 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.866219    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.867342 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.867314    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.867553 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.867533    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.867881 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.867862    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.868056 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.868040    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3deae563-f3ea-41cd-bd22-36ca90375806-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.888900 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.888866    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szphr\" (UniqueName: \"kubernetes.io/projected/5cb930ab-251e-4279-9963-f6d0d02b5b98-kube-api-access-szphr\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.889045 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.888875    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-svcsn\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:27.889045 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.888941    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3deae563-f3ea-41cd-bd22-36ca90375806-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.889267 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.889241    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64bss\" (UniqueName: \"kubernetes.io/projected/3deae563-f3ea-41cd-bd22-36ca90375806-kube-api-access-64bss\") pod \"maas-default-gateway-openshift-default-769df7846c-tpxsw\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:27.992666 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:27.992594    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:28.029459 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:28.029424    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:28.314428 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:28.314354    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"]
Oct 29 21:53:28.328850 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:53:28.328824    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3deae563_f3ea_41cd_bd22_36ca90375806.slice/crio-a3458f8a5d83dae0ba239155511aa81f3d6c6960ffe2bc0011640a95f89ed31a WatchSource:0}: Error finding container a3458f8a5d83dae0ba239155511aa81f3d6c6960ffe2bc0011640a95f89ed31a: Status 404 returned error can't find the container with id a3458f8a5d83dae0ba239155511aa81f3d6c6960ffe2bc0011640a95f89ed31a
Oct 29 21:53:28.331080 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:28.331052    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:53:28.331195 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:28.331115    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:53:28.331195 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:28.331169    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:53:28.343435 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:28.343410    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"]
Oct 29 21:53:28.344930 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:53:28.344893    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb930ab_251e_4279_9963_f6d0d02b5b98.slice/crio-161358df064570e9791834ba48c7d5644f5eb1050a8e3ae21d5293a0630ca5e5 WatchSource:0}: Error finding container 161358df064570e9791834ba48c7d5644f5eb1050a8e3ae21d5293a0630ca5e5: Status 404 returned error can't find the container with id 161358df064570e9791834ba48c7d5644f5eb1050a8e3ae21d5293a0630ca5e5
Oct 29 21:53:28.347219 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:28.347191    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:53:28.347306 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:28.347254    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:53:28.347306 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:28.347282    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:53:28.908167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:28.897511    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn" event={"ID":"5cb930ab-251e-4279-9963-f6d0d02b5b98","Type":"ContainerStarted","Data":"17ee9f1d9e0b7ac1bb816e877090da880f55c29ce36640fc62c157f18d7d9f07"}
Oct 29 21:53:28.908167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:28.897577    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn" event={"ID":"5cb930ab-251e-4279-9963-f6d0d02b5b98","Type":"ContainerStarted","Data":"161358df064570e9791834ba48c7d5644f5eb1050a8e3ae21d5293a0630ca5e5"}
Oct 29 21:53:28.908788 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:28.908323    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw" event={"ID":"3deae563-f3ea-41cd-bd22-36ca90375806","Type":"ContainerStarted","Data":"364b733813fa4052cb66474978426de8deaf019c517bcfd6dde7747b8aa7ddcb"}
Oct 29 21:53:28.908788 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:28.908388    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw" event={"ID":"3deae563-f3ea-41cd-bd22-36ca90375806","Type":"ContainerStarted","Data":"a3458f8a5d83dae0ba239155511aa81f3d6c6960ffe2bc0011640a95f89ed31a"}
Oct 29 21:53:28.950438 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:28.950385    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn" podStartSLOduration=1.9503719130000001 podStartE2EDuration="1.950371913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:28.949125122 +0000 UTC m=+425.106250640" watchObservedRunningTime="2026-10-29 21:53:28.950371913 +0000 UTC m=+425.107497411"
Oct 29 21:53:28.993187 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:28.992892    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:29.004817 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:29.004744    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw" podStartSLOduration=2.004723121 podStartE2EDuration="2.004723121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:29.001758279 +0000 UTC m=+425.158883777" watchObservedRunningTime="2026-10-29 21:53:29.004723121 +0000 UTC m=+425.161848619"
Oct 29 21:53:29.030728 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:29.030700    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:29.031955 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:29.031931    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-svcsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 29 21:53:29.032071 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:29.031973    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn" podUID="5cb930ab-251e-4279-9963-f6d0d02b5b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 29 21:53:29.996178 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:29.996012    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tpxsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 29 21:53:29.996178 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:29.996072    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw" podUID="3deae563-f3ea-41cd-bd22-36ca90375806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 29 21:53:30.030516 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:30.030480    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-svcsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 29 21:53:30.030673 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:30.030532    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn" podUID="5cb930ab-251e-4279-9963-f6d0d02b5b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 29 21:53:30.995176 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:30.994557    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tpxsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 29 21:53:30.995176 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:30.994622    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw" podUID="3deae563-f3ea-41cd-bd22-36ca90375806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 29 21:53:31.030375 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:31.030335    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-svcsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 29 21:53:31.030750 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:31.030391    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn" podUID="5cb930ab-251e-4279-9963-f6d0d02b5b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 29 21:53:32.011591 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:32.011554    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tpxsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 29 21:53:32.011921 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:32.011879    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw" podUID="3deae563-f3ea-41cd-bd22-36ca90375806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 29 21:53:32.030161 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:32.030109    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-svcsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 29 21:53:32.030353 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:32.030327    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn" podUID="5cb930ab-251e-4279-9963-f6d0d02b5b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 29 21:53:32.993655 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:32.993609    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tpxsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 29 21:53:32.994313 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:32.994281    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw" podUID="3deae563-f3ea-41cd-bd22-36ca90375806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 29 21:53:33.030264 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:33.030188    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-svcsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 29 21:53:33.030264 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:33.030245    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn" podUID="5cb930ab-251e-4279-9963-f6d0d02b5b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 29 21:53:33.707765 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:33.707728    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:53:33.995219 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:33.994739    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tpxsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 29 21:53:33.995219 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:33.994799    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw" podUID="3deae563-f3ea-41cd-bd22-36ca90375806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 29 21:53:34.030298 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:34.030268    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-svcsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 29 21:53:34.030450 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:34.030321    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn" podUID="5cb930ab-251e-4279-9963-f6d0d02b5b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 29 21:53:34.173122 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:34.173088    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:53:34.997372 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:34.997344    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:34.997738 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:34.997604    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:34.998248 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:34.998232    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 21:53:35.034477 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:35.034444    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:35.034742 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:35.034716    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:35.035333 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:35.035314    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 21:53:35.414494 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:35.414454    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:53:35.537528 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:35.537494    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:53:41.501545 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:53:41.501500    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:53:44.454712 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:44.454670    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:53:48.116743 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:48.116699    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:53:49.349396 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:49.349356    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:53:52.691391 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:52.691329    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:53:54.586982 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:54.586945    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:53:57.907120 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:53:57.907083    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:00.377924 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:00.377896    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:02.269369 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:02.269327    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:10.152201 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:10.152165    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:11.489395 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:54:11.489360    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:54:22.373531 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:22.373487    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:22.563319 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:22.563284    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:23.977712 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:23.977678    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:24.104567 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:24.104529    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:41.489209 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:54:41.489114    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:54:58.498819 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:58.498786    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:58.654462 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:58.654421    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:58.908899 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:58.908857    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:59.056327 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:59.056294    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:59.186033 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:59.185945    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:59.323302 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:59.323270    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:59.455519 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:59.455450    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:59.589899 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:59.589863    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:54:59.853470 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:54:59.853436    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:00.018548 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:00.018495    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:00.468342 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:00.468301    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:01.084783 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:01.084747    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:01.286879 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:01.286843    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:02.229047 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:02.229017    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:02.348811 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:02.348778    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:02.484893 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:02.484816    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:02.602327 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:02.602297    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:02.718943 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:02.718908    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:02.851687 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:02.851653    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:02.969793 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:02.969726    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:03.880065 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:03.880032    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:04.088339 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:04.088302    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:04.211168 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:04.210996    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:04.953259 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:04.953229    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:05.078964 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:05.078915    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:05.572948 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:05.572914    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:05.690298 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:05.690252    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:05.885304 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:05.885271    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:06.004631 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:06.004593    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:06.531935 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:06.531901    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:06.655499 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:06.655442    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:06.993513 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:06.993478    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:07.112663 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:07.112623    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:07.388300 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:07.388262    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:07.511433 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:07.511398    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:07.855295 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:07.855265    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:08.076038 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:08.076003    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:08.195018 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:08.194936    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:08.608653 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:08.608626    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:09.075828 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.075758    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:09.215574 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.215545    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:09.573348 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.573274    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"]
Oct 29 21:55:09.576399 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.576382    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:09.587282 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.587258    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gblp4\" (UniqueName: \"kubernetes.io/projected/7acb526b-7d51-450f-953b-56eeb030c102-kube-api-access-gblp4\") pod \"maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:09.587378 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.587312    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acb526b-7d51-450f-953b-56eeb030c102-openshift-service-ca\") pod \"maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:09.587478 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.587457    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-metrics-certs\") pod \"maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:09.587537 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.587522    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-maas-api-tls\") pod \"maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:09.602666 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.602595    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-de7186ef-serving-cert\""
Oct 29 21:55:09.605877 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.605844    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"]
Oct 29 21:55:09.665978 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.665941    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:09.688373 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.688339    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acb526b-7d51-450f-953b-56eeb030c102-openshift-service-ca\") pod \"maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:09.688545 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.688432    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-metrics-certs\") pod \"maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:09.688545 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.688497    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-maas-api-tls\") pod \"maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:09.688545 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.688527    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gblp4\" (UniqueName: \"kubernetes.io/projected/7acb526b-7d51-450f-953b-56eeb030c102-kube-api-access-gblp4\") pod \"maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:09.688721 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:55:09.688667    4550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-serving-cert: secret "maas-api-e2e-ait-de7186ef-serving-cert" not found
Oct 29 21:55:09.688826 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:55:09.688808    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-maas-api-tls podName:7acb526b-7d51-450f-953b-56eeb030c102 nodeName:}" failed. No retries permitted until 2026-10-29 21:55:10.188753752 +0000 UTC m=+526.345879243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-maas-api-tls") pod "maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z" (UID: "7acb526b-7d51-450f-953b-56eeb030c102") : secret "maas-api-e2e-ait-de7186ef-serving-cert" not found
Oct 29 21:55:09.689167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.689119    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acb526b-7d51-450f-953b-56eeb030c102-openshift-service-ca\") pod \"maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:09.692692 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.692664    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-metrics-certs\") pod \"maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:09.712571 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.712538    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gblp4\" (UniqueName: \"kubernetes.io/projected/7acb526b-7d51-450f-953b-56eeb030c102-kube-api-access-gblp4\") pod \"maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:09.790860 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.790828    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:09.912874 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:09.912843    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:10.192882 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:10.192787    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-maas-api-tls\") pod \"maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:10.195206 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:10.195185    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-maas-api-tls\") pod \"maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:10.292781 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:10.292751    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:10.489387 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:10.489292    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:10.791761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:10.791734    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"]
Oct 29 21:55:10.793900 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:55:10.793871    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7acb526b_7d51_450f_953b_56eeb030c102.slice/crio-aaead16530b11e1529342f3e23b14fa8d67d598d923e69b28a99829ee234af7f WatchSource:0}: Error finding container aaead16530b11e1529342f3e23b14fa8d67d598d923e69b28a99829ee234af7f: Status 404 returned error can't find the container with id aaead16530b11e1529342f3e23b14fa8d67d598d923e69b28a99829ee234af7f
Oct 29 21:55:11.430941 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.430903    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z" event={"ID":"7acb526b-7d51-450f-953b-56eeb030c102","Type":"ContainerStarted","Data":"772a0e45c2d25e291f06f0c470f75856acd867499772c140bc79fff31279ba9e"}
Oct 29 21:55:11.431365 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.430983    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z" event={"ID":"7acb526b-7d51-450f-953b-56eeb030c102","Type":"ContainerStarted","Data":"aaead16530b11e1529342f3e23b14fa8d67d598d923e69b28a99829ee234af7f"}
Oct 29 21:55:11.431365 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.431058    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:11.474419 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.474374    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z" podStartSLOduration=2.032122962 podStartE2EDuration="2.474361289s" totalImagesPullingTime="442.238327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:11.473411566 +0000 UTC m=+527.630537083" watchObservedRunningTime="2026-10-29 21:55:11.474361289 +0000 UTC m=+527.631486787"
Oct 29 21:55:11.489486 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:55:11.489418    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:55:11.495158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.495110    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"]
Oct 29 21:55:11.498825 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.498810    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:11.504406 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.504380    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-maas-api-tls\") pod \"maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:11.504501 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.504442    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-metrics-certs\") pod \"maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:11.504588 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.504571    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4w4k\" (UniqueName: \"kubernetes.io/projected/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-kube-api-access-v4w4k\") pod \"maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:11.504657 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.504614    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-openshift-service-ca\") pod \"maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:11.516451 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.516431    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f97668b9-serving-cert\""
Oct 29 21:55:11.522081 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.522060    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"]
Oct 29 21:55:11.605877 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.605766    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4w4k\" (UniqueName: \"kubernetes.io/projected/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-kube-api-access-v4w4k\") pod \"maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:11.605877 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.605824    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-openshift-service-ca\") pod \"maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:11.606220 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.605912    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-maas-api-tls\") pod \"maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:11.606220 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.605971    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-metrics-certs\") pod \"maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:11.606471 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:55:11.606444    4550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-serving-cert: secret "maas-api-e2e-ait-f97668b9-serving-cert" not found
Oct 29 21:55:11.606607 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:55:11.606507    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-maas-api-tls podName:3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82 nodeName:}" failed. No retries permitted until 2026-10-29 21:55:12.106491299 +0000 UTC m=+528.263616787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-maas-api-tls") pod "maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6" (UID: "3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82") : secret "maas-api-e2e-ait-f97668b9-serving-cert" not found
Oct 29 21:55:11.606818 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.606794    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-openshift-service-ca\") pod \"maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:11.608500 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.608476    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-metrics-certs\") pod \"maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:11.613841 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.613819    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:11.630082 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.630061    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4w4k\" (UniqueName: \"kubernetes.io/projected/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-kube-api-access-v4w4k\") pod \"maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:11.791625 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:11.791545    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:12.110163 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:12.110112    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-maas-api-tls\") pod \"maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:12.112619 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:12.112597    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-maas-api-tls\") pod \"maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:12.412109 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:12.412024    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:12.717639 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:12.717560    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"]
Oct 29 21:55:12.721120 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:55:12.721094    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2bbbd0_a9ae_49df_93e1_dcd831cf2b82.slice/crio-fded4102f543fb144a866bbf24f9e299e2d5daeefd185bee18022935b78cf580 WatchSource:0}: Error finding container fded4102f543fb144a866bbf24f9e299e2d5daeefd185bee18022935b78cf580: Status 404 returned error can't find the container with id fded4102f543fb144a866bbf24f9e299e2d5daeefd185bee18022935b78cf580
Oct 29 21:55:13.243855 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:13.243812    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:13.365394 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:13.365350    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:13.441776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:13.441741    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6" event={"ID":"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82","Type":"ContainerStarted","Data":"b9295faab2571a30c7476a704a7e0edea4bb50532d3e1dd96c0ac2f8e29d287a"}
Oct 29 21:55:13.441920 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:13.441795    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6" event={"ID":"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82","Type":"ContainerStarted","Data":"fded4102f543fb144a866bbf24f9e299e2d5daeefd185bee18022935b78cf580"}
Oct 29 21:55:13.441920 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:13.441823    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:13.485922 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:13.485877    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6" podStartSLOduration=2.16641997 podStartE2EDuration="2.485863515s" totalImagesPullingTime="319.443545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:13.485601302 +0000 UTC m=+529.642726820" watchObservedRunningTime="2026-10-29 21:55:13.485863515 +0000 UTC m=+529.642989012"
Oct 29 21:55:13.597479 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:13.597448    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:15.371342 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:15.371302    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:15.492569 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:15.492532    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:16.950555 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:16.950518    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"]
Oct 29 21:55:16.954411 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:16.954386    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:16.977602 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:16.977576    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-e4e24e-serving-cert\""
Oct 29 21:55:16.987002 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:16.986974    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"]
Oct 29 21:55:17.048976 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.048946    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:17.057358 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.057335    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:17.057489 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.057400    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-metrics-certs\") pod \"maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:17.057532 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.057512    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nsqp\" (UniqueName: \"kubernetes.io/projected/a3b59718-adb6-437a-8c5d-3003d93b5203-kube-api-access-9nsqp\") pod \"maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:17.057567 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.057541    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b59718-adb6-437a-8c5d-3003d93b5203-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:17.158982 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.158949    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nsqp\" (UniqueName: \"kubernetes.io/projected/a3b59718-adb6-437a-8c5d-3003d93b5203-kube-api-access-9nsqp\") pod \"maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:17.159128 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.158986    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b59718-adb6-437a-8c5d-3003d93b5203-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:17.159128 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.159010    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:17.159269 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:55:17.159137    4550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-serving-cert: secret "maas-api-e2e-worker-w0-e4e24e-serving-cert" not found
Oct 29 21:55:17.159269 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:55:17.159210    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-maas-api-tls podName:a3b59718-adb6-437a-8c5d-3003d93b5203 nodeName:}" failed. No retries permitted until 2026-10-29 21:55:17.659190896 +0000 UTC m=+533.816316389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-maas-api-tls") pod "maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc" (UID: "a3b59718-adb6-437a-8c5d-3003d93b5203") : secret "maas-api-e2e-worker-w0-e4e24e-serving-cert" not found
Oct 29 21:55:17.159269 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.159224    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-metrics-certs\") pod \"maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:17.159722 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.159702    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b59718-adb6-437a-8c5d-3003d93b5203-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:17.161710 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.161687    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-metrics-certs\") pod \"maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:17.186438 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.186401    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nsqp\" (UniqueName: \"kubernetes.io/projected/a3b59718-adb6-437a-8c5d-3003d93b5203-kube-api-access-9nsqp\") pod \"maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:17.238246 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.238169    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:17.402965 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.402931    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:17.442511 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.442483    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:17.665216 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.665183    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:17.667844 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.667821    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:17.867000 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:17.866940    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:18.023419 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:18.023381    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:18.139416 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:18.139379    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:18.189624 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:18.189566    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"]
Oct 29 21:55:18.466160 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:18.466064    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc" event={"ID":"a3b59718-adb6-437a-8c5d-3003d93b5203","Type":"ContainerStarted","Data":"306d8b182e876cb7b9f94a1d621893c040112d53403752a17b4231a654134fa6"}
Oct 29 21:55:19.023415 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:19.023369    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:19.452707 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:19.452680    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:19.472385 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:19.472356    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc" event={"ID":"a3b59718-adb6-437a-8c5d-3003d93b5203","Type":"ContainerStarted","Data":"a9fd0940145c0102548abf2f08c2f56edd9ddf7789449e0494275813404b88ac"}
Oct 29 21:55:19.472385 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:19.472409    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:19.541390 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:19.541341    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc" podStartSLOduration=3.236802333 podStartE2EDuration="3.541324524s" totalImagesPullingTime="304.522191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:19.540542785 +0000 UTC m=+535.697668304" watchObservedRunningTime="2026-10-29 21:55:19.541324524 +0000 UTC m=+535.698450021"
Oct 29 21:55:20.907021 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:20.906985    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:22.755039 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:22.755002    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:24.540610 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:24.540573    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:25.482723 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:25.482695    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 21:55:26.316834 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:26.316800    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:26.436856 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:26.436819    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:26.566805 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:26.566762    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:28.233404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:28.233375    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:28.783719 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:28.783684    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:28.917969 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:28.917925    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:29.979028 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:29.978998    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:31.787804 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:31.787775    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:33.036872 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:33.036840    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:33.153284 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:33.153238    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:33.276756 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:33.276712    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:33.842187 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:33.842150    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:35.345086 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:35.345051    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:35.477371 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:35.477340    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:35.598276 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:35.598186    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:35.723934 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:35.723900    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:35.847280 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:35.847247    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:35.967603 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:35.967530    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:36.096705 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:36.096670    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:36.633841 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:36.633802    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:37.114924 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.114882    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:37.371008 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.370930    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"]
Oct 29 21:55:37.376435 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.376412    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.385188 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.385133    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:37.396564 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.396531    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"]
Oct 29 21:55:37.406218 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.406189    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"openshift-service-ca.crt\""
Oct 29 21:55:37.406443 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.406420    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"kube-root-ca.crt\""
Oct 29 21:55:37.406540 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.406498    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"default-dockercfg-8bczv\""
Oct 29 21:55:37.406599 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.406590    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"simulator-w0-e4e24e-kserve-self-signed-certs\""
Oct 29 21:55:37.456112 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.456072    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5mbb\" (UniqueName: \"kubernetes.io/projected/1021a25d-cebc-4bad-965d-6afc5bad66f9-kube-api-access-p5mbb\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.456288 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.456244    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-dshm\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.456348 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.456332    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-home\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.456396 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.456362    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1021a25d-cebc-4bad-965d-6afc5bad66f9-tls-certs\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.456432 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.456394    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-model-cache\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.456498 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.456485    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-tmp-dir\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.501660 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.501623    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:37.557628 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.557593    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-dshm\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.557802 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.557648    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-home\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.557802 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.557665    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1021a25d-cebc-4bad-965d-6afc5bad66f9-tls-certs\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.557802 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.557683    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-model-cache\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.557802 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.557700    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-tmp-dir\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.557802 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.557763    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5mbb\" (UniqueName: \"kubernetes.io/projected/1021a25d-cebc-4bad-965d-6afc5bad66f9-kube-api-access-p5mbb\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.558120 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.558096    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-home\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.558232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.558180    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-model-cache\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.558296 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.558249    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-tmp-dir\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.560010 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.559966    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-dshm\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.560236 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.560217    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1021a25d-cebc-4bad-965d-6afc5bad66f9-tls-certs\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.582138 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.582108    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5mbb\" (UniqueName: \"kubernetes.io/projected/1021a25d-cebc-4bad-965d-6afc5bad66f9-kube-api-access-p5mbb\") pod \"simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") " pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:37.689571 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:37.689486    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:38.003285 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:38.003263    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"]
Oct 29 21:55:38.005214 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:55:38.005185    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1021a25d_cebc_4bad_965d_6afc5bad66f9.slice/crio-4d7bb4b6938344b2d996cb6ee38c8323402ffbdde74a73a099ad66f8d65f5b82 WatchSource:0}: Error finding container 4d7bb4b6938344b2d996cb6ee38c8323402ffbdde74a73a099ad66f8d65f5b82: Status 404 returned error can't find the container with id 4d7bb4b6938344b2d996cb6ee38c8323402ffbdde74a73a099ad66f8d65f5b82
Oct 29 21:55:38.566686 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:38.566647    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" event={"ID":"1021a25d-cebc-4bad-965d-6afc5bad66f9","Type":"ContainerStarted","Data":"4d7bb4b6938344b2d996cb6ee38c8323402ffbdde74a73a099ad66f8d65f5b82"}
Oct 29 21:55:38.592849 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:38.592812    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:38.978619 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:38.978583    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:39.106423 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:39.106317    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:39.222558 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:39.222527    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:39.579576 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:39.579539    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" event={"ID":"1021a25d-cebc-4bad-965d-6afc5bad66f9","Type":"ContainerStarted","Data":"3815e3e8034638f89db5397b31ff7d098d778f110ac796b6c24806ccdd49e40b"}
Oct 29 21:55:39.623279 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:39.623225    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podStartSLOduration=1.383718054 podStartE2EDuration="2.623208188s" totalImagesPullingTime="1.239490134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:39.621731263 +0000 UTC m=+555.778856786" watchObservedRunningTime="2026-10-29 21:55:39.623208188 +0000 UTC m=+555.780333689"
Oct 29 21:55:39.690454 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:39.690426    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:39.781435 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:39.781399    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:39.913679 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:39.913598    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:40.685845 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:40.685805    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:40.999735 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:40.999615    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:41.125283 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:41.125251    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:41.241295 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:41.241257    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:41.425782 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:41.425744    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:41.490534 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:55:41.490501    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:55:42.797572 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:42.797535    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:42.923511 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:42.923465    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:43.044791 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:43.044754    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:43.160374 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:43.160342    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:43.439518 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:43.439439    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:43.832720 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:43.832688    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:44.051555 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:44.051359    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:44.181683 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:44.181547    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:44.314667 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:44.314635    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:44.431096 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:44.431061    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:45.079688 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:45.079657    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:46.368162 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.368111    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"]
Oct 29 21:55:46.368656 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.368378    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z" podUID="7acb526b-7d51-450f-953b-56eeb030c102" containerName="maas-api" containerID="cri-o://772a0e45c2d25e291f06f0c470f75856acd867499772c140bc79fff31279ba9e" gracePeriod=30
Oct 29 21:55:46.438988 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.438959    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:46.617708 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.617672    4550 generic.go:356] "Generic (PLEG): container finished" podID="7acb526b-7d51-450f-953b-56eeb030c102" containerID="772a0e45c2d25e291f06f0c470f75856acd867499772c140bc79fff31279ba9e" exitCode=0
Oct 29 21:55:46.617894 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.617768    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z" event={"ID":"7acb526b-7d51-450f-953b-56eeb030c102","Type":"ContainerDied","Data":"772a0e45c2d25e291f06f0c470f75856acd867499772c140bc79fff31279ba9e"}
Oct 29 21:55:46.617894 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.617815    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z" event={"ID":"7acb526b-7d51-450f-953b-56eeb030c102","Type":"ContainerDied","Data":"aaead16530b11e1529342f3e23b14fa8d67d598d923e69b28a99829ee234af7f"}
Oct 29 21:55:46.617894 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.617842    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaead16530b11e1529342f3e23b14fa8d67d598d923e69b28a99829ee234af7f"
Oct 29 21:55:46.627003 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.626943    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:46.721648 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.721620    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"]
Oct 29 21:55:46.721843 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.721816    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6" podUID="3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82" containerName="maas-api" containerID="cri-o://b9295faab2571a30c7476a704a7e0edea4bb50532d3e1dd96c0ac2f8e29d287a" gracePeriod=30
Oct 29 21:55:46.751046 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.751021    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-maas-api-tls\") pod \"7acb526b-7d51-450f-953b-56eeb030c102\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") "
Oct 29 21:55:46.751183 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.751167    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7acb526b-7d51-450f-953b-56eeb030c102-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acb526b-7d51-450f-953b-56eeb030c102-openshift-service-ca\") pod \"7acb526b-7d51-450f-953b-56eeb030c102\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") "
Oct 29 21:55:46.751252 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.751206    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-metrics-certs\") pod \"7acb526b-7d51-450f-953b-56eeb030c102\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") "
Oct 29 21:55:46.751312 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.751272    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7acb526b-7d51-450f-953b-56eeb030c102-kube-api-access-gblp4\" (UniqueName: \"kubernetes.io/projected/7acb526b-7d51-450f-953b-56eeb030c102-kube-api-access-gblp4\") pod \"7acb526b-7d51-450f-953b-56eeb030c102\" (UID: \"7acb526b-7d51-450f-953b-56eeb030c102\") "
Oct 29 21:55:46.752037 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.752012    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7acb526b-7d51-450f-953b-56eeb030c102-openshift-service-ca" pod "7acb526b-7d51-450f-953b-56eeb030c102" (UID: "7acb526b-7d51-450f-953b-56eeb030c102"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:46.753699 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.753676    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7acb526b-7d51-450f-953b-56eeb030c102-kube-api-access-gblp4" pod "7acb526b-7d51-450f-953b-56eeb030c102" (UID: "7acb526b-7d51-450f-953b-56eeb030c102"). InnerVolumeSpecName "kube-api-access-gblp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:46.754131 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.754113    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-maas-api-tls" pod "7acb526b-7d51-450f-953b-56eeb030c102" (UID: "7acb526b-7d51-450f-953b-56eeb030c102"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:55:46.754285 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.754268    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-metrics-certs" pod "7acb526b-7d51-450f-953b-56eeb030c102" (UID: "7acb526b-7d51-450f-953b-56eeb030c102"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:55:46.790833 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.790803    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:46.852692 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.852659    4550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acb526b-7d51-450f-953b-56eeb030c102-openshift-service-ca\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:55:46.852813 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.852695    4550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-metrics-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:55:46.852813 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.852711    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gblp4\" (UniqueName: \"kubernetes.io/projected/7acb526b-7d51-450f-953b-56eeb030c102-kube-api-access-gblp4\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:55:46.852813 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.852724    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7acb526b-7d51-450f-953b-56eeb030c102-maas-api-tls\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:55:46.908770 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.908738    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:46.978668 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:46.978644    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:47.036291 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.036252    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:47.054617 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.054586    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-kube-api-access-v4w4k\" (UniqueName: \"kubernetes.io/projected/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-kube-api-access-v4w4k\") pod \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") "
Oct 29 21:55:47.054756 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.054671    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-metrics-certs\") pod \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") "
Oct 29 21:55:47.054880 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.054860    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-maas-api-tls\") pod \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") "
Oct 29 21:55:47.054948 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.054903    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-openshift-service-ca\") pod \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\" (UID: \"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82\") "
Oct 29 21:55:47.055400 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.055355    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-openshift-service-ca" pod "3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82" (UID: "3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:47.056899 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.056871    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-kube-api-access-v4w4k" pod "3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82" (UID: "3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82"). InnerVolumeSpecName "kube-api-access-v4w4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:47.057014 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.056907    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-maas-api-tls" pod "3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82" (UID: "3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:55:47.057123 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.057098    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-metrics-certs" pod "3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82" (UID: "3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:55:47.156078 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.155998    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4w4k\" (UniqueName: \"kubernetes.io/projected/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-kube-api-access-v4w4k\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:55:47.156078 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.156032    4550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-metrics-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:55:47.156078 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.156046    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-maas-api-tls\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:55:47.156078 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.156061    4550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82-openshift-service-ca\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 21:55:47.238970 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.238936    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:47.416487 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.416410    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:47.535384 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.535338    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:47.623985 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.623950    4550 generic.go:356] "Generic (PLEG): container finished" podID="3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82" containerID="b9295faab2571a30c7476a704a7e0edea4bb50532d3e1dd96c0ac2f8e29d287a" exitCode=0
Oct 29 21:55:47.624127 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.624007    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"
Oct 29 21:55:47.624127 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.624035    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6" event={"ID":"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82","Type":"ContainerDied","Data":"b9295faab2571a30c7476a704a7e0edea4bb50532d3e1dd96c0ac2f8e29d287a"}
Oct 29 21:55:47.624127 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.624095    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6" event={"ID":"3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82","Type":"ContainerDied","Data":"fded4102f543fb144a866bbf24f9e299e2d5daeefd185bee18022935b78cf580"}
Oct 29 21:55:47.624127 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.624121    4550 scope.go:122] "RemoveContainer" containerID="b9295faab2571a30c7476a704a7e0edea4bb50532d3e1dd96c0ac2f8e29d287a"
Oct 29 21:55:47.624353 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.624291    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"
Oct 29 21:55:47.634680 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.634658    4550 scope.go:122] "RemoveContainer" containerID="b9295faab2571a30c7476a704a7e0edea4bb50532d3e1dd96c0ac2f8e29d287a"
Oct 29 21:55:47.634998 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:55:47.634969    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9295faab2571a30c7476a704a7e0edea4bb50532d3e1dd96c0ac2f8e29d287a\": container with ID starting with b9295faab2571a30c7476a704a7e0edea4bb50532d3e1dd96c0ac2f8e29d287a not found: ID does not exist" containerID="b9295faab2571a30c7476a704a7e0edea4bb50532d3e1dd96c0ac2f8e29d287a"
Oct 29 21:55:47.635095 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.635005    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9295faab2571a30c7476a704a7e0edea4bb50532d3e1dd96c0ac2f8e29d287a"} err="failed to get container status \"b9295faab2571a30c7476a704a7e0edea4bb50532d3e1dd96c0ac2f8e29d287a\": rpc error: code = NotFound desc = could not find container \"b9295faab2571a30c7476a704a7e0edea4bb50532d3e1dd96c0ac2f8e29d287a\": container with ID starting with b9295faab2571a30c7476a704a7e0edea4bb50532d3e1dd96c0ac2f8e29d287a not found: ID does not exist"
Oct 29 21:55:47.675336 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.675268    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"]
Oct 29 21:55:47.687775 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.687749    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f97668b9-7779d79c58-rfhg6"]
Oct 29 21:55:47.690073 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.690051    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:47.702957 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.702934    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:47.718817 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.718793    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"]
Oct 29 21:55:47.722155 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.722123    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-de7186ef-7bc5bdf55-nvt2z"]
Oct 29 21:55:47.979922 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.979842    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:47.987980 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.987955    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82" path="/var/lib/kubelet/pods/3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82/volumes"
Oct 29 21:55:47.988403 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:47.988388    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7acb526b-7d51-450f-953b-56eeb030c102" path="/var/lib/kubelet/pods/7acb526b-7d51-450f-953b-56eeb030c102/volumes"
Oct 29 21:55:48.257645 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:48.257551    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:48.386988 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:48.386956    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:48.508791 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:48.508702    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:48.642270 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:48.642241    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 21:55:48.715962 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:48.715916    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:48.846280 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:48.846249    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:48.970909 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:48.970870    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:49.098334 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:49.098252    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:49.492451 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:49.492369    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:49.626241 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:49.626209    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:49.766206 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:49.766119    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:50.495710 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:50.495674    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:50.622634 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:50.622596    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:50.751783 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:50.751687    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:50.869699 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:50.869657    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:51.179547 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:51.179509    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:51.717231 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:51.717194    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:51.835342 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:51.835306    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:51.951641 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:51.951608    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:52.074366 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:52.074330    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:52.679651 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:52.679614    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:52.794467 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:52.794426    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:52.911944 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:52.911917    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:53.380441 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:53.380406    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:53.495829 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:53.495793    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:53.664965 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:53.664871    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:53.791664 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:53.791629    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:53.920507 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:53.920427    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:54.428724 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:54.428686    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:54.541813 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:54.541778    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:55.580937 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:55.580895    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:55.774501 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:55.774453    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:55.892673 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:55.892591    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:56.923458 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:56.923417    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:57.039606 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:57.039569    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:57.510439 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:57.510394    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:57.631322 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:57.631287    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:57.760076 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:57.760047    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:59.194945 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.194912    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"]
Oct 29 21:55:59.195502 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.195482    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7acb526b-7d51-450f-953b-56eeb030c102" containerName="maas-api"
Oct 29 21:55:59.195543 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.195506    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7acb526b-7d51-450f-953b-56eeb030c102" containerName="maas-api"
Oct 29 21:55:59.195578 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.195557    4550 cpu_manager.go:414] "containerMap: removing container" podUID="3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82" containerName="maas-api"
Oct 29 21:55:59.195578 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.195566    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82" containerName="maas-api"
Oct 29 21:55:59.195658 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.195646    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a2bbbd0-a9ae-49df-93e1-dcd831cf2b82" containerName="maas-api"
Oct 29 21:55:59.195694 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.195666    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7acb526b-7d51-450f-953b-56eeb030c102" containerName="maas-api"
Oct 29 21:55:59.200833 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.200808    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.220951 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.220844    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"]
Oct 29 21:55:59.231380 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.231353    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"premium-w0-e4e24e-kserve-self-signed-certs\""
Oct 29 21:55:59.273675 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.273642    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:59.375078 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.375035    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-tmp-dir\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.375282 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.375198    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-home\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.375282 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.375229    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-dshm\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.375425 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.375360    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-model-cache\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.375425 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.375384    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/280967ca-516c-40d1-891f-7c3edb621486-tls-certs\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.375425 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.375406    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz72m\" (UniqueName: \"kubernetes.io/projected/280967ca-516c-40d1-891f-7c3edb621486-kube-api-access-tz72m\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.392161 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.392116    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:59.476255 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.476178    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-tmp-dir\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.476255 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.476249    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-home\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.476416 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.476267    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-dshm\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.476416 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.476349    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-model-cache\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.476416 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.476365    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/280967ca-516c-40d1-891f-7c3edb621486-tls-certs\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.476416 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.476394    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz72m\" (UniqueName: \"kubernetes.io/projected/280967ca-516c-40d1-891f-7c3edb621486-kube-api-access-tz72m\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.476616 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.476599    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-tmp-dir\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.476681 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.476663    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-home\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.476734 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.476714    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-model-cache\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.478568 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.478536    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-dshm\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.478838 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.478821    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/280967ca-516c-40d1-891f-7c3edb621486-tls-certs\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.504938 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.504911    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz72m\" (UniqueName: \"kubernetes.io/projected/280967ca-516c-40d1-891f-7c3edb621486-kube-api-access-tz72m\") pod \"premium-w0-e4e24e-kserve-56954b7d4f-7vf8t\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") " pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.512959 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.512934    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:55:59.513062 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.512929    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:55:59.855923 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:55:59.855897    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"]
Oct 29 21:55:59.857886 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:55:59.857863    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod280967ca_516c_40d1_891f_7c3edb621486.slice/crio-97912b1ee22297ed53851b26f56e9f4e283eb7930fbef51eab462d7f62f78615 WatchSource:0}: Error finding container 97912b1ee22297ed53851b26f56e9f4e283eb7930fbef51eab462d7f62f78615: Status 404 returned error can't find the container with id 97912b1ee22297ed53851b26f56e9f4e283eb7930fbef51eab462d7f62f78615
Oct 29 21:56:00.693561 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:00.693522    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" event={"ID":"280967ca-516c-40d1-891f-7c3edb621486","Type":"ContainerStarted","Data":"77c0db2ce0493dce7314d756465856372e7f0dc2e7c68e4490e7cd5d4c83f89b"}
Oct 29 21:56:00.693965 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:00.693571    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" event={"ID":"280967ca-516c-40d1-891f-7c3edb621486","Type":"ContainerStarted","Data":"97912b1ee22297ed53851b26f56e9f4e283eb7930fbef51eab462d7f62f78615"}
Oct 29 21:56:00.737222 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:00.737174    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podStartSLOduration=1.737161355 podStartE2EDuration="1.737161355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:00.735883213 +0000 UTC m=+576.893008712" watchObservedRunningTime="2026-10-29 21:56:00.737161355 +0000 UTC m=+576.894286847"
Oct 29 21:56:01.210278 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:01.210225    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:01.349039 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:01.349004    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:01.470893 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:01.470795    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:01.513842 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:01.513808    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:56:03.192642 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:03.192597    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:04.873058 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:04.873022    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"]
Oct 29 21:56:04.879740 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:04.879712    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:04.897291 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:04.897263    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-743a5e-serving-cert\""
Oct 29 21:56:04.903351 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:04.902202    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"]
Oct 29 21:56:04.926644 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:04.926601    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2shpm\" (UniqueName: \"kubernetes.io/projected/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-kube-api-access-2shpm\") pod \"maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:04.926795 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:04.926722    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:04.926850 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:04.926789    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-maas-api-tls\") pod \"maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:04.926850 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:04.926832    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-metrics-certs\") pod \"maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:04.960200 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:04.960169    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:05.027868 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:05.027836    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-metrics-certs\") pod \"maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:05.028050 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:05.027979    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2shpm\" (UniqueName: \"kubernetes.io/projected/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-kube-api-access-2shpm\") pod \"maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:05.028050 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:05.028024    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:05.028238 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:05.028068    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-maas-api-tls\") pod \"maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:05.028849 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:05.028826    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:05.030585 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:05.030560    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-maas-api-tls\") pod \"maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:05.030671 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:05.030615    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-metrics-certs\") pod \"maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:05.052105 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:05.052076    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2shpm\" (UniqueName: \"kubernetes.io/projected/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-kube-api-access-2shpm\") pod \"maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:05.079611 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:05.079571    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:05.196425 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:05.196339    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:05.201015 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:05.200976    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:05.505400 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:05.505374    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"]
Oct 29 21:56:05.507349 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:56:05.507320    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c47f2b9_9ea7_44aa_92ad_1a4e38511a5d.slice/crio-28fe4867adfb455654109acdd5d0803a9a975aa3d668b4a996839369dd37ba66 WatchSource:0}: Error finding container 28fe4867adfb455654109acdd5d0803a9a975aa3d668b4a996839369dd37ba66: Status 404 returned error can't find the container with id 28fe4867adfb455654109acdd5d0803a9a975aa3d668b4a996839369dd37ba66
Oct 29 21:56:05.724603 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:05.724563    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96" event={"ID":"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d","Type":"ContainerStarted","Data":"28fe4867adfb455654109acdd5d0803a9a975aa3d668b4a996839369dd37ba66"}
Oct 29 21:56:06.150248 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:06.150216    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:06.615229 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:06.615130    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:06.730766 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:06.730728    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96" event={"ID":"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d","Type":"ContainerStarted","Data":"d2b097890345faba1a69a3aedcdcf93c381f7102b836e5cc1454bf2b0cf2c1da"}
Oct 29 21:56:06.730944 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:06.730790    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:06.754985 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:06.754955    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:06.775732 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:06.775677    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96" podStartSLOduration=2.325427102 podStartE2EDuration="2.775662591s" totalImagesPullingTime="450.235489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:06.774444744 +0000 UTC m=+582.931570242" watchObservedRunningTime="2026-10-29 21:56:06.775662591 +0000 UTC m=+582.932788088"
Oct 29 21:56:06.871987 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:06.871907    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:06.994505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:06.994474    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:08.414202 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:08.414125    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:09.513602 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:09.513570    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:56:09.526540 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:09.526511    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:56:09.758779 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:09.758746    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 21:56:09.978923 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:09.978891    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"]
Oct 29 21:56:09.983846 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:09.983820    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:10.010428 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.010393    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"]
Oct 29 21:56:10.010772 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.010749    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-8a9e5f-serving-cert\""
Oct 29 21:56:10.078656 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.078624    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf254b1-d374-4b55-8fec-d04f6b6501c1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:10.078837 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.078664    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf254b1-d374-4b55-8fec-d04f6b6501c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:10.078837 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.078683    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-528gl\" (UniqueName: \"kubernetes.io/projected/8cf254b1-d374-4b55-8fec-d04f6b6501c1-kube-api-access-528gl\") pod \"maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:10.079029 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.078994    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf254b1-d374-4b55-8fec-d04f6b6501c1-metrics-certs\") pod \"maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:10.080167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.080114    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:10.180158 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.180105    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf254b1-d374-4b55-8fec-d04f6b6501c1-metrics-certs\") pod \"maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:10.180339 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.180255    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf254b1-d374-4b55-8fec-d04f6b6501c1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:10.180339 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.180296    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf254b1-d374-4b55-8fec-d04f6b6501c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:10.180339 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.180315    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528gl\" (UniqueName: \"kubernetes.io/projected/8cf254b1-d374-4b55-8fec-d04f6b6501c1-kube-api-access-528gl\") pod \"maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:10.181067 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.181043    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf254b1-d374-4b55-8fec-d04f6b6501c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:10.182630 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.182612    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf254b1-d374-4b55-8fec-d04f6b6501c1-metrics-certs\") pod \"maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:10.182722 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.182660    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf254b1-d374-4b55-8fec-d04f6b6501c1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:10.201999 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.201969    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:10.204122 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.204097    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-528gl\" (UniqueName: \"kubernetes.io/projected/8cf254b1-d374-4b55-8fec-d04f6b6501c1-kube-api-access-528gl\") pod \"maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:10.302703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.302617    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:10.324611 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.324580    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:10.450410 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.450364    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:10.616072 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.616044    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"]
Oct 29 21:56:10.618153 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:56:10.618110    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf254b1_d374_4b55_8fec_d04f6b6501c1.slice/crio-d37c2c8d601b073a276f963cf41634921c515e3811f7210df6dccb80722fae03 WatchSource:0}: Error finding container d37c2c8d601b073a276f963cf41634921c515e3811f7210df6dccb80722fae03: Status 404 returned error can't find the container with id d37c2c8d601b073a276f963cf41634921c515e3811f7210df6dccb80722fae03
Oct 29 21:56:10.754071 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.754034    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5" event={"ID":"8cf254b1-d374-4b55-8fec-d04f6b6501c1","Type":"ContainerStarted","Data":"d37c2c8d601b073a276f963cf41634921c515e3811f7210df6dccb80722fae03"}
Oct 29 21:56:10.872441 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:10.872314    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:11.491173 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:56:11.491113    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:56:11.761863 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:11.761775    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5" event={"ID":"8cf254b1-d374-4b55-8fec-d04f6b6501c1","Type":"ContainerStarted","Data":"09090d2c0196811d1d5c4e28d40ca3cb4980753df4964c5a25d4a1b98216f97b"}
Oct 29 21:56:11.762348 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:11.761904    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:11.807668 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:11.807615    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5" podStartSLOduration=2.493245548 podStartE2EDuration="2.807602049s" totalImagesPullingTime="314.356501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:11.806188597 +0000 UTC m=+587.963314092" watchObservedRunningTime="2026-10-29 21:56:11.807602049 +0000 UTC m=+587.964727546"
Oct 29 21:56:11.933322 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:11.933286    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:12.742282 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:12.742247    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 21:56:13.722762 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:13.722729    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:13.842524 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:13.842474    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:14.327590 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:14.327536    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:14.454972 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:14.454932    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:14.581038 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:14.580944    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:15.696107 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:15.696067    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:17.323180 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:17.323128    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:17.772796 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:17.772769    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 21:56:19.026586 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:19.026546    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:20.626438 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:20.626398    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:22.507248 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.507212    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"]
Oct 29 21:56:22.511241 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.511213    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.539037 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.539012    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a3c02e\"/\"kube-root-ca.crt\""
Oct 29 21:56:22.539326 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.539294    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a3c02e\"/\"openshift-service-ca.crt\""
Oct 29 21:56:22.539409 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.539395    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a3c02e\"/\"default-dockercfg-5krsk\""
Oct 29 21:56:22.540981 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.540960    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"]
Oct 29 21:56:22.542000 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.541983    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a3c02e\"/\"premium-w1-a3c02e-kserve-self-signed-certs\""
Oct 29 21:56:22.547253 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.547233    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:22.625349 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.625314    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4090c710-5f67-4aff-b19c-d02e2347cf77-tls-certs\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.625506 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.625364    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-model-cache\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.625506 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.625476    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-home\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.625581 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.625541    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-dshm\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.625581 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.625570    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v2rn\" (UniqueName: \"kubernetes.io/projected/4090c710-5f67-4aff-b19c-d02e2347cf77-kube-api-access-2v2rn\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.625679 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.625666    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-tmp-dir\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.703878 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.703830    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:22.727052 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.727018    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-dshm\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.727234 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.727062    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v2rn\" (UniqueName: \"kubernetes.io/projected/4090c710-5f67-4aff-b19c-d02e2347cf77-kube-api-access-2v2rn\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.727234 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.727134    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-tmp-dir\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.727234 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.727178    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4090c710-5f67-4aff-b19c-d02e2347cf77-tls-certs\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.727234 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.727200    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-model-cache\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.727435 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.727340    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-home\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.727610 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.727590    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-model-cache\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.727687 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.727622    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-tmp-dir\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.727808 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.727768    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-home\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.729423 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.729404    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-dshm\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.729579 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.729563    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4090c710-5f67-4aff-b19c-d02e2347cf77-tls-certs\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.751919 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.751895    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v2rn\" (UniqueName: \"kubernetes.io/projected/4090c710-5f67-4aff-b19c-d02e2347cf77-kube-api-access-2v2rn\") pod \"premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") " pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.824749 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.824724    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:22.876219 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:22.876131    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:23.127669 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:23.127647    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"]
Oct 29 21:56:23.129641 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:56:23.129613    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4090c710_5f67_4aff_b19c_d02e2347cf77.slice/crio-3fd4614d47e8c8eade32a2f6d2564e973860bc2dd850c699ec7b816fe399c023 WatchSource:0}: Error finding container 3fd4614d47e8c8eade32a2f6d2564e973860bc2dd850c699ec7b816fe399c023: Status 404 returned error can't find the container with id 3fd4614d47e8c8eade32a2f6d2564e973860bc2dd850c699ec7b816fe399c023
Oct 29 21:56:23.478966 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:23.478886    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:23.606247 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:23.606195    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:23.741367 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:23.741236    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:23.822838 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:23.822806    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" event={"ID":"4090c710-5f67-4aff-b19c-d02e2347cf77","Type":"ContainerStarted","Data":"06bd3eb66bf2649d44f36649e534bfd32f3bee597d1fa4bad669f526a5fa015e"}
Oct 29 21:56:23.823035 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:23.822847    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" event={"ID":"4090c710-5f67-4aff-b19c-d02e2347cf77","Type":"ContainerStarted","Data":"3fd4614d47e8c8eade32a2f6d2564e973860bc2dd850c699ec7b816fe399c023"}
Oct 29 21:56:23.825057 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:23.825045    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:23.877239 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:23.877179    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podStartSLOduration=1.877164895 podStartE2EDuration="1.877164895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:23.875317994 +0000 UTC m=+600.032443493" watchObservedRunningTime="2026-10-29 21:56:23.877164895 +0000 UTC m=+600.034290387"
Oct 29 21:56:23.892055 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:23.892015    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:24.372690 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:24.372653    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:24.521454 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:24.521414    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:24.647005 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:24.646905    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:24.990292 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:24.990214    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"]
Oct 29 21:56:24.996621 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:24.996591    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.011844 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.011816    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b31f3f\"/\"kube-root-ca.crt\""
Oct 29 21:56:25.011980 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.011874    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b31f3f\"/\"openshift-service-ca.crt\""
Oct 29 21:56:25.017588 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.017560    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"]
Oct 29 21:56:25.024712 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.024686    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b31f3f\"/\"simulator-w4-b31f3f-kserve-self-signed-certs\""
Oct 29 21:56:25.024837 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.024795    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b31f3f\"/\"default-dockercfg-6tf66\""
Oct 29 21:56:25.048379 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.048342    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:25.159857 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.159814    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2bzq\" (UniqueName: \"kubernetes.io/projected/660c3442-f3d1-4eb4-a013-232f82f8d81b-kube-api-access-b2bzq\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.160041 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.159873    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-dshm\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.160041 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.159965    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-tmp-dir\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.160213 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.160069    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-home\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.160213 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.160124    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660c3442-f3d1-4eb4-a013-232f82f8d81b-tls-certs\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.160375 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.160227    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-model-cache\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.260818 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.260740    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-dshm\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.260818 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.260770    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-tmp-dir\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.260818 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.260813    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-home\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.261107 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.260836    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660c3442-f3d1-4eb4-a013-232f82f8d81b-tls-certs\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.261107 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.260873    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-model-cache\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.261107 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.260945    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2bzq\" (UniqueName: \"kubernetes.io/projected/660c3442-f3d1-4eb4-a013-232f82f8d81b-kube-api-access-b2bzq\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.261331 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.261292    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-home\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.261382 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.261336    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-model-cache\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.261382 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.261368    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-tmp-dir\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.263218 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.263197    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-dshm\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.263412 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.263395    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660c3442-f3d1-4eb4-a013-232f82f8d81b-tls-certs\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.284615 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.284589    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2bzq\" (UniqueName: \"kubernetes.io/projected/660c3442-f3d1-4eb4-a013-232f82f8d81b-kube-api-access-b2bzq\") pod \"simulator-w4-b31f3f-kserve-7884846896-8npx9\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") " pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.311947 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.311911    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:25.630623 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.630584    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"]
Oct 29 21:56:25.632005 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:56:25.631981    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod660c3442_f3d1_4eb4_a013_232f82f8d81b.slice/crio-57254b36415c8cfec25e58715d953ce903b7c2ddf7d1fd20fe06df5ec83f4ca9 WatchSource:0}: Error finding container 57254b36415c8cfec25e58715d953ce903b7c2ddf7d1fd20fe06df5ec83f4ca9: Status 404 returned error can't find the container with id 57254b36415c8cfec25e58715d953ce903b7c2ddf7d1fd20fe06df5ec83f4ca9
Oct 29 21:56:25.836926 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.836878    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" event={"ID":"660c3442-f3d1-4eb4-a013-232f82f8d81b","Type":"ContainerStarted","Data":"c74a06680102af036ec7a72f5e0b5442dc7967ff80d2fb5ff0c3f3ff06f161c9"}
Oct 29 21:56:25.837454 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.836941    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" event={"ID":"660c3442-f3d1-4eb4-a013-232f82f8d81b","Type":"ContainerStarted","Data":"57254b36415c8cfec25e58715d953ce903b7c2ddf7d1fd20fe06df5ec83f4ca9"}
Oct 29 21:56:25.880815 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:25.880697    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podStartSLOduration=1.8806824629999999 podStartE2EDuration="1.880682463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:25.879498555 +0000 UTC m=+602.036624054" watchObservedRunningTime="2026-10-29 21:56:25.880682463 +0000 UTC m=+602.037807961"
Oct 29 21:56:26.116527 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:26.116494    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:26.312327 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:26.312248    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:27.664678 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:27.664644    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:27.917463 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:27.917388    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:28.281774 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:28.281677    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:28.411563 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:28.411533    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:28.539669 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:28.539580    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:29.722289 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:29.722253    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:31.336195 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.336157    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"]
Oct 29 21:56:31.340414 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.340390    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.359426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.359404    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e4e24e-openshift-default-dockercfg-9t52r\""
Oct 29 21:56:31.375517 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.375483    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"]
Oct 29 21:56:31.424462 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.424434    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a89ba55-981b-475e-a86c-833c547e65d5-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.424614 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.424498    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a89ba55-981b-475e-a86c-833c547e65d5-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.424614 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.424578    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.424757 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.424611    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a89ba55-981b-475e-a86c-833c547e65d5-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.424757 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.424639    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.424837 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.424771    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkj5d\" (UniqueName: \"kubernetes.io/projected/5a89ba55-981b-475e-a86c-833c547e65d5-kube-api-access-vkj5d\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.424891 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.424857    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.424953 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.424896    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.425014 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.424977    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.525853 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.525822    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.526007 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.525891    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a89ba55-981b-475e-a86c-833c547e65d5-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.526007 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.525960    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a89ba55-981b-475e-a86c-833c547e65d5-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.526007 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.526001    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.526217 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.526032    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a89ba55-981b-475e-a86c-833c547e65d5-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.526217 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.526062    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.526217 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.526124    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkj5d\" (UniqueName: \"kubernetes.io/projected/5a89ba55-981b-475e-a86c-833c547e65d5-kube-api-access-vkj5d\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.526373 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.526208    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.526427 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.526403    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.526483 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.526267    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.526628 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.526606    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.526755 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.526653    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.526814 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.526783    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.527034 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.527011    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a89ba55-981b-475e-a86c-833c547e65d5-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.528667 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.528645    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.529274 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.529256    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a89ba55-981b-475e-a86c-833c547e65d5-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.550719 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.550698    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkj5d\" (UniqueName: \"kubernetes.io/projected/5a89ba55-981b-475e-a86c-833c547e65d5-kube-api-access-vkj5d\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.550830 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.550811    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a89ba55-981b-475e-a86c-833c547e65d5-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.601372 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.601281    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:31.653897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.653874    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:31.972171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.972115    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"]
Oct 29 21:56:31.973487 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:56:31.973461    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a89ba55_981b_475e_a86c_833c547e65d5.slice/crio-05e30f1e115ac6f46cedffa4711a8777d71976bc84478f2cb33a06de6b7eb519 WatchSource:0}: Error finding container 05e30f1e115ac6f46cedffa4711a8777d71976bc84478f2cb33a06de6b7eb519: Status 404 returned error can't find the container with id 05e30f1e115ac6f46cedffa4711a8777d71976bc84478f2cb33a06de6b7eb519
Oct 29 21:56:31.982083 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.976586    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:56:31.982083 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.976743    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:56:31.982083 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:31.976835    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:56:32.825644 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:32.825608    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:32.838356 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:32.838332    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:32.892452 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:32.892414    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6" event={"ID":"5a89ba55-981b-475e-a86c-833c547e65d5","Type":"ContainerStarted","Data":"377137e92fa9ed2894a0b7a6a0c0d53b0710ecedb11b2d868728a49a99017b88"}
Oct 29 21:56:32.892656 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:32.892482    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6" event={"ID":"5a89ba55-981b-475e-a86c-833c547e65d5","Type":"ContainerStarted","Data":"05e30f1e115ac6f46cedffa4711a8777d71976bc84478f2cb33a06de6b7eb519"}
Oct 29 21:56:32.904040 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:32.904016    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 21:56:32.940593 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:32.940533    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6" podStartSLOduration=1.9405191450000001 podStartE2EDuration="1.940519145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:32.93913089 +0000 UTC m=+609.096256388" watchObservedRunningTime="2026-10-29 21:56:32.940519145 +0000 UTC m=+609.097644643"
Oct 29 21:56:33.351539 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:33.351507    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:33.655163 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:33.655072    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:33.656507 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:33.656474    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 29 21:56:33.656643 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:33.656531    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6" podUID="5a89ba55-981b-475e-a86c-833c547e65d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 29 21:56:34.490423 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:34.490224    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:34.655028 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:34.654993    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 29 21:56:34.655028 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:34.655071    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6" podUID="5a89ba55-981b-475e-a86c-833c547e65d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 29 21:56:35.147787 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:35.147752    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:35.312749 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:35.312719    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:35.325933 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:35.325907    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:35.655012 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:35.654980    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 29 21:56:35.655413 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:35.655030    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6" podUID="5a89ba55-981b-475e-a86c-833c547e65d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 29 21:56:35.920847 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:35.920765    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 21:56:36.654237 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:36.654197    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 29 21:56:36.654409 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:36.654251    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6" podUID="5a89ba55-981b-475e-a86c-833c547e65d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 29 21:56:36.938251 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:36.938175    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:37.654868 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:37.654837    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 29 21:56:37.655043 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:37.654891    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6" podUID="5a89ba55-981b-475e-a86c-833c547e65d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 29 21:56:38.610216 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:38.610179    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:38.798075 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:38.798042    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:38.798499 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:38.798466    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:38.801754 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:38.801733    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 21:56:40.222013 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:40.221965    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:41.353183 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.337132    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"]
Oct 29 21:56:41.371960 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.371921    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.372165 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.372040    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"]
Oct 29 21:56:41.398213 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.398180    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:41.399065 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.399039    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-743a5e\"/\"openshift-service-ca.crt\""
Oct 29 21:56:41.399335 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.399314    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-743a5e\"/\"simulator-w5-743a5e-kserve-self-signed-certs\""
Oct 29 21:56:41.399416 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.399408    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-743a5e\"/\"default-dockercfg-skx5w\""
Oct 29 21:56:41.399518 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.399502    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-743a5e\"/\"kube-root-ca.crt\""
Oct 29 21:56:41.460094 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.460056    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-tmp-dir\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.460271 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.460227    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-model-cache\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.460339 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.460271    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g8kc\" (UniqueName: \"kubernetes.io/projected/09bace6e-b116-4c71-a2c1-1265a83b1608-kube-api-access-8g8kc\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.460396 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.460371    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-dshm\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.460472 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.460450    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09bace6e-b116-4c71-a2c1-1265a83b1608-tls-certs\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.460531 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.460514    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-home\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.504592 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:56:41.504540    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:56:41.561230 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.561188    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-model-cache\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.561431 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.561253    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g8kc\" (UniqueName: \"kubernetes.io/projected/09bace6e-b116-4c71-a2c1-1265a83b1608-kube-api-access-8g8kc\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.561431 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.561292    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-dshm\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.561431 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.561338    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09bace6e-b116-4c71-a2c1-1265a83b1608-tls-certs\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.561431 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.561366    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-home\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.561663 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.561438    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-tmp-dir\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.561725 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.561680    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-model-cache\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.562258 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.562114    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-tmp-dir\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.562258 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.562208    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-home\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.568522 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.567760    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09bace6e-b116-4c71-a2c1-1265a83b1608-tls-certs\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.568522 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.567936    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-dshm\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.588573 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.588499    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g8kc\" (UniqueName: \"kubernetes.io/projected/09bace6e-b116-4c71-a2c1-1265a83b1608-kube-api-access-8g8kc\") pod \"simulator-w5-743a5e-kserve-796d7465f-c889r\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") " pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.697049 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.696945    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:41.953577 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:41.953491    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:42.008029 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:42.007983    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"]
Oct 29 21:56:42.009074 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:56:42.009046    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09bace6e_b116_4c71_a2c1_1265a83b1608.slice/crio-4f25640d63cba2ea1a06a65aa2c1f771782af08a3f2e3879ffc2a397c975e61e WatchSource:0}: Error finding container 4f25640d63cba2ea1a06a65aa2c1f771782af08a3f2e3879ffc2a397c975e61e: Status 404 returned error can't find the container with id 4f25640d63cba2ea1a06a65aa2c1f771782af08a3f2e3879ffc2a397c975e61e
Oct 29 21:56:42.073376 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:42.073339    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:42.950284 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:42.950243    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" event={"ID":"09bace6e-b116-4c71-a2c1-1265a83b1608","Type":"ContainerStarted","Data":"5737bb8f15a270c47dfcf47588883fc8bb33ab86e947b008275217751d4bb0fc"}
Oct 29 21:56:42.950688 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:42.950293    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" event={"ID":"09bace6e-b116-4c71-a2c1-1265a83b1608","Type":"ContainerStarted","Data":"4f25640d63cba2ea1a06a65aa2c1f771782af08a3f2e3879ffc2a397c975e61e"}
Oct 29 21:56:42.952394 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:42.952370    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:43.000705 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:43.000649    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podStartSLOduration=2.000634777 podStartE2EDuration="2.000634777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:42.999940167 +0000 UTC m=+619.157065667" watchObservedRunningTime="2026-10-29 21:56:43.000634777 +0000 UTC m=+619.157760295"
Oct 29 21:56:43.515629 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:43.515593    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:43.697129 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:43.697093    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:45.105392 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:45.105365    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:46.705365 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:46.705330    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:48.271162 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:48.271102    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:49.923267 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:49.923230    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:50.233628 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.233539    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"]
Oct 29 21:56:50.238979 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.238957    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.255908 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.255880    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a940bc\"/\"kube-root-ca.crt\""
Oct 29 21:56:50.268064 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.268032    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a940bc\"/\"openshift-service-ca.crt\""
Oct 29 21:56:50.268241 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.268076    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a940bc\"/\"premium-w6-a940bc-kserve-self-signed-certs\""
Oct 29 21:56:50.268352 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.268046    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a940bc\"/\"default-dockercfg-mcsgh\""
Oct 29 21:56:50.271284 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.271263    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"]
Oct 29 21:56:50.300951 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.300922    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:50.350553 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.350515    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-model-cache\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.350713 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.350638    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-home\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.350713 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.350700    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-tmp-dir\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.350815 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.350736    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvhwf\" (UniqueName: \"kubernetes.io/projected/cd727a21-2709-44be-a77d-f8cb1482e317-kube-api-access-xvhwf\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.350903 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.350879    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd727a21-2709-44be-a77d-f8cb1482e317-tls-certs\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.350940 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.350926    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-dshm\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.452227 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.452184    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-tmp-dir\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.452430 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.452264    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvhwf\" (UniqueName: \"kubernetes.io/projected/cd727a21-2709-44be-a77d-f8cb1482e317-kube-api-access-xvhwf\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.452430 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.452315    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd727a21-2709-44be-a77d-f8cb1482e317-tls-certs\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.452430 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.452348    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-dshm\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.452430 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.452403    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-model-cache\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.452681 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.452498    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-home\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.452738 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.452701    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-tmp-dir\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.452910 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.452863    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-home\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.453042 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.452966    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-model-cache\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.455288 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.455264    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-dshm\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.455656 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.455634    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd727a21-2709-44be-a77d-f8cb1482e317-tls-certs\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.477196 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.477157    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvhwf\" (UniqueName: \"kubernetes.io/projected/cd727a21-2709-44be-a77d-f8cb1482e317-kube-api-access-xvhwf\") pod \"premium-w6-a940bc-kserve-75f5c84cdc-s7jk8\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") " pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.553761 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.553671    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:50.859740 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.859517    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"]
Oct 29 21:56:50.861989 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:56:50.861963    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd727a21_2709_44be_a77d_f8cb1482e317.slice/crio-7e1c2349f02ae92beb37a4bbb3966677cd85de9fbc532c2aca5b71dc445db7a6 WatchSource:0}: Error finding container 7e1c2349f02ae92beb37a4bbb3966677cd85de9fbc532c2aca5b71dc445db7a6: Status 404 returned error can't find the container with id 7e1c2349f02ae92beb37a4bbb3966677cd85de9fbc532c2aca5b71dc445db7a6
Oct 29 21:56:50.995404 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.995368    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" event={"ID":"cd727a21-2709-44be-a77d-f8cb1482e317","Type":"ContainerStarted","Data":"e6c233bb309b03bd1102aed7dda96acefac85e501b016c87a934c5dd2e7540eb"}
Oct 29 21:56:50.995810 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:50.995412    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" event={"ID":"cd727a21-2709-44be-a77d-f8cb1482e317","Type":"ContainerStarted","Data":"7e1c2349f02ae92beb37a4bbb3966677cd85de9fbc532c2aca5b71dc445db7a6"}
Oct 29 21:56:51.039028 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:51.038974    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podStartSLOduration=1.038958695 podStartE2EDuration="1.038958695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:51.038438615 +0000 UTC m=+627.195564112" watchObservedRunningTime="2026-10-29 21:56:51.038958695 +0000 UTC m=+627.196084227"
Oct 29 21:56:51.554265 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:51.554233    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:56:51.622597 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:51.622559    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:51.697911 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:51.697878    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:51.711362 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:51.711333    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:52.010940 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:52.010911    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 21:56:53.261760 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:53.261706    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:54.379852 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.379817    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"]
Oct 29 21:56:54.383582 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.383560    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.400525 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.400506    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a3c02e-openshift-default-dockercfg-tvljv\""
Oct 29 21:56:54.414639 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.414616    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"]
Oct 29 21:56:54.495232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.495196    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.495400 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.495234    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.495400 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.495267    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jlqk\" (UniqueName: \"kubernetes.io/projected/45d0f73f-8e77-46d0-99e4-9101e92a1f13-kube-api-access-9jlqk\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.495400 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.495375    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.495543 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.495480    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.495543 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.495540    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.495633 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.495560    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.495633 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.495585    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.495732 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.495690    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.596470 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.596438    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.596470 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.596472    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.596733 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.596492    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jlqk\" (UniqueName: \"kubernetes.io/projected/45d0f73f-8e77-46d0-99e4-9101e92a1f13-kube-api-access-9jlqk\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.596733 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.596509    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.596733 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.596702    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.596902 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.596782    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.596902 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.596808    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.596902 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.596844    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.596902 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.596877    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.597088 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.596973    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.597088 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.596990    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.597088 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.597066    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.597265 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.597242    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.597804 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.597783    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.598974 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.598950    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.599388 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.599360    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.620288 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.620266    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.620421 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.620405    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jlqk\" (UniqueName: \"kubernetes.io/projected/45d0f73f-8e77-46d0-99e4-9101e92a1f13-kube-api-access-9jlqk\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:54.696943 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:54.696863    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:55.019550 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:55.019525    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"]
Oct 29 21:56:55.020974 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:56:55.020942    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d0f73f_8e77_46d0_99e4_9101e92a1f13.slice/crio-0e8968a0b5919756b52570cbcf16fcbd1884b671f9c254fa00a89e9584830267 WatchSource:0}: Error finding container 0e8968a0b5919756b52570cbcf16fcbd1884b671f9c254fa00a89e9584830267: Status 404 returned error can't find the container with id 0e8968a0b5919756b52570cbcf16fcbd1884b671f9c254fa00a89e9584830267
Oct 29 21:56:55.023172 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:55.023113    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:56:55.023245 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:55.023223    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:56:55.023285 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:55.023256    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:56:55.074847 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:55.074810    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:56.023234 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:56.023199    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m" event={"ID":"45d0f73f-8e77-46d0-99e4-9101e92a1f13","Type":"ContainerStarted","Data":"9edf2c81a4d9c7bbd2d4783bc99591247d91dbe8ff4e68c98f78347b8e76cd0d"}
Oct 29 21:56:56.023631 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:56.023255    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m" event={"ID":"45d0f73f-8e77-46d0-99e4-9101e92a1f13","Type":"ContainerStarted","Data":"0e8968a0b5919756b52570cbcf16fcbd1884b671f9c254fa00a89e9584830267"}
Oct 29 21:56:56.102573 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:56.102502    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m" podStartSLOduration=2.102479654 podStartE2EDuration="2.102479654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:56.087819898 +0000 UTC m=+632.244945396" watchObservedRunningTime="2026-10-29 21:56:56.102479654 +0000 UTC m=+632.259605156"
Oct 29 21:56:56.697216 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:56.697187    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:56:56.698578 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:56.698545    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 29 21:56:56.698813 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:56.698784    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m" podUID="45d0f73f-8e77-46d0-99e4-9101e92a1f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 29 21:56:56.859957 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:56.859851    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:57.697964 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:57.697925    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 29 21:56:57.698487 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:57.697975    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m" podUID="45d0f73f-8e77-46d0-99e4-9101e92a1f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 29 21:56:58.509741 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:58.509667    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:56:58.698583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:58.698214    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 29 21:56:58.698583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:58.698274    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m" podUID="45d0f73f-8e77-46d0-99e4-9101e92a1f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 29 21:56:59.697612 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:59.697571    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 29 21:56:59.697785 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:56:59.697631    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m" podUID="45d0f73f-8e77-46d0-99e4-9101e92a1f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 29 21:57:00.196778 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:00.196747    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:00.554700 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:00.554606    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:57:00.588645 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:00.588603    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:57:00.697827 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:00.697790    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 29 21:57:00.698005 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:00.697845    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m" podUID="45d0f73f-8e77-46d0-99e4-9101e92a1f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 29 21:57:01.084540 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:01.084515    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 21:57:01.820434 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:01.820386    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:01.973245 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:01.973200    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:57:01.974058 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:01.974027    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:57:01.975043 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:01.975021    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 21:57:03.379832 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:03.379802    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:04.925703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:04.925672    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:06.523536 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:06.523503    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:08.226528 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:08.226460    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:09.160941 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.160908    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"]
Oct 29 21:57:09.176897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.176854    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.185582 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.185553    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"]
Oct 29 21:57:09.194868 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.194843    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a9def7\"/\"default-dockercfg-26fhs\""
Oct 29 21:57:09.195008 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.194926    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a9def7\"/\"premium-w2-a9def7-kserve-self-signed-certs\""
Oct 29 21:57:09.204328 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.204275    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a9def7\"/\"kube-root-ca.crt\""
Oct 29 21:57:09.204328 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.204275    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a9def7\"/\"openshift-service-ca.crt\""
Oct 29 21:57:09.222726 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.222689    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:09.361412 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.361365    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-model-cache\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.361917 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.361492    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-tls-certs\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.361917 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.361536    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-home\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.361917 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.361648    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-dshm\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.361917 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.361698    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-tmp-dir\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.361917 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.361794    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv9dc\" (UniqueName: \"kubernetes.io/projected/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-kube-api-access-gv9dc\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.463450 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.463350    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-dshm\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.463450 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.463412    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-tmp-dir\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.463684 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.463487    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv9dc\" (UniqueName: \"kubernetes.io/projected/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-kube-api-access-gv9dc\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.463684 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.463539    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-model-cache\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.463684 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.463595    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-tls-certs\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.463684 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.463634    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-home\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.464652 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.464081    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-home\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.464652 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.464385    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-model-cache\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.464652 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.464616    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-tmp-dir\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.467491 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.467466    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-dshm\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.468972 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.468948    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-tls-certs\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.490705 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.490675    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv9dc\" (UniqueName: \"kubernetes.io/projected/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-kube-api-access-gv9dc\") pod \"premium-w2-a9def7-kserve-cb8565594-9r87m\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") " pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.490849 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.490835    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:09.809926 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:09.809900    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"]
Oct 29 21:57:09.811516 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:57:09.811482    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ec7f0f1_9454_46d6_991e_3dfb6d2645da.slice/crio-7b6ac936e14d3d68f3c08ef13e4c39332d8440e48610ca865b3af5cfdfb927e5 WatchSource:0}: Error finding container 7b6ac936e14d3d68f3c08ef13e4c39332d8440e48610ca865b3af5cfdfb927e5: Status 404 returned error can't find the container with id 7b6ac936e14d3d68f3c08ef13e4c39332d8440e48610ca865b3af5cfdfb927e5
Oct 29 21:57:10.024163 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:10.024100    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:10.110918 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:10.110881    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" event={"ID":"2ec7f0f1-9454-46d6-991e-3dfb6d2645da","Type":"ContainerStarted","Data":"4440beae683b0a246401270560cc8144dc3de7301f46f3257e57fbe18932e54e"}
Oct 29 21:57:10.111332 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:10.111297    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" event={"ID":"2ec7f0f1-9454-46d6-991e-3dfb6d2645da","Type":"ContainerStarted","Data":"7b6ac936e14d3d68f3c08ef13e4c39332d8440e48610ca865b3af5cfdfb927e5"}
Oct 29 21:57:10.154835 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:10.154787    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podStartSLOduration=1.154772603 podStartE2EDuration="1.154772603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:10.153482983 +0000 UTC m=+646.310608481" watchObservedRunningTime="2026-10-29 21:57:10.154772603 +0000 UTC m=+646.311898101"
Oct 29 21:57:10.491816 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:10.491744    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:10.728777 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:10.728743    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:11.493561 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:57:11.493525    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:57:11.910794 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:11.910754    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:13.591660 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:13.591628    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:15.121909 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:15.121882    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:16.725910 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:16.725870    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:18.411111 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:18.411071    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:18.678628 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:18.678537    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:19.162258 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:19.162228    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:19.491365 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:19.491285    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:19.503945 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:19.503923    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:20.183047 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:20.183015    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 21:57:20.228947 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:20.228915    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:21.836768 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:21.836728    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:21.893571 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:21.893535    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"]
Oct 29 21:57:21.897187 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:21.897131    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:21.925882 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:21.925838    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8a9e5f\"/\"openshift-service-ca.crt\""
Oct 29 21:57:21.925882 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:21.925874    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a9e5f\"/\"default-dockercfg-q247w\""
Oct 29 21:57:21.926203 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:21.926182    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8a9e5f\"/\"kube-root-ca.crt\""
Oct 29 21:57:21.928331 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:21.928305    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"]
Oct 29 21:57:21.931239 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:21.931217    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a9e5f\"/\"premium-w3-8a9e5f-kserve-self-signed-certs\""
Oct 29 21:57:21.973945 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:21.973885    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:21.978991 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:21.978958    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-dshm\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:21.979117 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:21.979088    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-tmp-dir\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:21.979117 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:21.979108    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff86f8c-7f09-43cb-bffc-889eb6e56181-tls-certs\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:21.979263 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:21.979132    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmdk2\" (UniqueName: \"kubernetes.io/projected/eff86f8c-7f09-43cb-bffc-889eb6e56181-kube-api-access-lmdk2\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:21.979306 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:21.979275    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-model-cache\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:21.979306 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:21.979301    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-home\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:22.080910 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:22.080873    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-tmp-dir\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:22.081580 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:22.081555    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff86f8c-7f09-43cb-bffc-889eb6e56181-tls-certs\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:22.081719 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:22.081601    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmdk2\" (UniqueName: \"kubernetes.io/projected/eff86f8c-7f09-43cb-bffc-889eb6e56181-kube-api-access-lmdk2\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:22.081719 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:22.081681    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-model-cache\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:22.081719 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:22.081715    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-home\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:22.081877 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:22.081830    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-dshm\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:22.085809 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:22.081512    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-tmp-dir\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:22.085809 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:22.084660    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-dshm\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:22.091738 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:22.091380    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff86f8c-7f09-43cb-bffc-889eb6e56181-tls-certs\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:22.091738 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:22.091410    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-home\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:22.091738 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:22.091671    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-model-cache\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:22.113081 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:22.113041    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmdk2\" (UniqueName: \"kubernetes.io/projected/eff86f8c-7f09-43cb-bffc-889eb6e56181-kube-api-access-lmdk2\") pod \"premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") " pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:22.210989 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:22.210957    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:22.516921 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:22.516894    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"]
Oct 29 21:57:22.518762 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:57:22.518730    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeff86f8c_7f09_43cb_bffc_889eb6e56181.slice/crio-20b586f6057b1b7f7d62123d13bba7b56ddc497372aca004507ac8bda20fcff3 WatchSource:0}: Error finding container 20b586f6057b1b7f7d62123d13bba7b56ddc497372aca004507ac8bda20fcff3: Status 404 returned error can't find the container with id 20b586f6057b1b7f7d62123d13bba7b56ddc497372aca004507ac8bda20fcff3
Oct 29 21:57:23.188736 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:23.188700    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" event={"ID":"eff86f8c-7f09-43cb-bffc-889eb6e56181","Type":"ContainerStarted","Data":"746dc9186f20b1de1da56f58ea4219aed87a1c0d2e7cf50914e2e548e36f8751"}
Oct 29 21:57:23.189236 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:23.188744    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" event={"ID":"eff86f8c-7f09-43cb-bffc-889eb6e56181","Type":"ContainerStarted","Data":"20b586f6057b1b7f7d62123d13bba7b56ddc497372aca004507ac8bda20fcff3"}
Oct 29 21:57:23.211411 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:23.211377    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:23.233056 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:23.233010    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podStartSLOduration=2.2329930669999998 podStartE2EDuration="2.232993067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:23.231640973 +0000 UTC m=+659.388766493" watchObservedRunningTime="2026-10-29 21:57:23.232993067 +0000 UTC m=+659.390118583"
Oct 29 21:57:23.530288 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:23.530210    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:25.223523 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:25.223487    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:26.817995 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:26.817962    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:28.561113 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:28.561061    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:30.124951 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:30.124917    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:30.247914 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:30.247878    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:31.819127 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:31.819081    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:32.211582 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:32.211549    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:32.224315 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:32.224293    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:32.254944 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:32.254920    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 21:57:33.453509 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:33.453474    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:35.121825 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:35.121786    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:36.835913 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:36.835827    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:38.741366 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:38.741331    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:40.433869 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:40.433836    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:41.489627 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:57:41.489593    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:57:42.126823 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:42.126773    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:43.810277 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:43.810237    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:45.490316 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:45.490283    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:47.057890 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:47.057858    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:47.249540 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:47.249502    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:48.763232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:48.763192    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:50.320532 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:50.320502    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:51.933725 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:51.933691    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:53.511873 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:53.511832    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:54.565838 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.565804    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"]
Oct 29 21:57:54.569725 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.569706    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.605541 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.605516    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"]
Oct 29 21:57:54.624707 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.624673    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.624838 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.624744    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.624910 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.624849    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.624910 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.624884    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.624910 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.624907    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.625074 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.625036    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpk87\" (UniqueName: \"kubernetes.io/projected/e32c602e-d9fc-4a01-93f1-39391f91653d-kube-api-access-hpk87\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.625130 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.625072    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.625130 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.625094    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.625260 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.625205    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32c602e-d9fc-4a01-93f1-39391f91653d-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.726240 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.726213    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.726382 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.726294    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.726382 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.726324    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.726382 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.726346    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.726635 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.726384    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpk87\" (UniqueName: \"kubernetes.io/projected/e32c602e-d9fc-4a01-93f1-39391f91653d-kube-api-access-hpk87\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.726635 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.726502    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.726635 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.726549    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.726635 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.726585    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32c602e-d9fc-4a01-93f1-39391f91653d-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.726635 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.726622    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.726894 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.726672    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.726894 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.726865    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.727001 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.726960    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.727060 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.727021    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.727386 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.727362    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32c602e-d9fc-4a01-93f1-39391f91653d-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.728739 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.728716    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.728818 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.728803    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.750024 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.749995    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpk87\" (UniqueName: \"kubernetes.io/projected/e32c602e-d9fc-4a01-93f1-39391f91653d-kube-api-access-hpk87\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.750165 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.750093    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:54.882676 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:54.882654    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:55.204750 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:55.204724    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"]
Oct 29 21:57:55.206374 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:57:55.206347    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32c602e_d9fc_4a01_93f1_39391f91653d.slice/crio-78c1a8f137a70455b96f426e7cb160835961e052adcab0a286c4d7b58aa9c8eb WatchSource:0}: Error finding container 78c1a8f137a70455b96f426e7cb160835961e052adcab0a286c4d7b58aa9c8eb: Status 404 returned error can't find the container with id 78c1a8f137a70455b96f426e7cb160835961e052adcab0a286c4d7b58aa9c8eb
Oct 29 21:57:55.208705 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:55.208673    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:57:55.208705 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:55.208689    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:55.208829 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:55.208739    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:57:55.208829 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:55.208768    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:57:55.371126 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:55.371093    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f" event={"ID":"e32c602e-d9fc-4a01-93f1-39391f91653d","Type":"ContainerStarted","Data":"dd69ccf940b46f52d608a13a960546f6ce5b31dd1db7f1b712871f57a20825dd"}
Oct 29 21:57:55.371302 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:55.371159    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f" event={"ID":"e32c602e-d9fc-4a01-93f1-39391f91653d","Type":"ContainerStarted","Data":"78c1a8f137a70455b96f426e7cb160835961e052adcab0a286c4d7b58aa9c8eb"}
Oct 29 21:57:55.417453 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:55.417402    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f" podStartSLOduration=1.41738685 podStartE2EDuration="1.41738685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:55.416477268 +0000 UTC m=+691.573602767" watchObservedRunningTime="2026-10-29 21:57:55.41738685 +0000 UTC m=+691.574512349"
Oct 29 21:57:55.885167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:55.884244    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:57:56.886832 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:56.886793    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 21:57:56.887246 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:56.886858    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f" podUID="e32c602e-d9fc-4a01-93f1-39391f91653d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 21:57:57.087547 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:57.087514    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:57.886597 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:57.886540    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 21:57:57.886809 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:57.886594    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f" podUID="e32c602e-d9fc-4a01-93f1-39391f91653d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 21:57:58.749207 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:58.749126    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:57:58.887609 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:58.887563    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 21:57:58.891498 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:58.891450    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f" podUID="e32c602e-d9fc-4a01-93f1-39391f91653d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 21:57:59.723110 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.723077    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"]
Oct 29 21:57:59.727202 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.727184    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.742468 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.742446    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b31f3f-openshift-default-dockercfg-ddh7z\""
Oct 29 21:57:59.756703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.756676    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"]
Oct 29 21:57:59.802691 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.802656    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2913dd75-7d51-4ee4-b705-f6a7396d75de-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.802843 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.802700    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.802843 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.802786    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.802843 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.802834    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.802961 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.802870    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9rbr\" (UniqueName: \"kubernetes.io/projected/2913dd75-7d51-4ee4-b705-f6a7396d75de-kube-api-access-c9rbr\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.803021 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.802991    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.803152 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.803127    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.803228 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.803215    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.803272 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.803240    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.886507 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.886471    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 21:57:59.886689 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.886521    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f" podUID="e32c602e-d9fc-4a01-93f1-39391f91653d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 21:57:59.903941 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.903907    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.904084 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.904001    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.904084 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.904051    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.904084 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.904072    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.904268 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.904096    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2913dd75-7d51-4ee4-b705-f6a7396d75de-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.904268 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.904117    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.904268 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.904136    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.904268 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.904205    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.904268 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.904235    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9rbr\" (UniqueName: \"kubernetes.io/projected/2913dd75-7d51-4ee4-b705-f6a7396d75de-kube-api-access-c9rbr\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.904563 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.904531    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.904608 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.904561    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.904822 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.904793    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2913dd75-7d51-4ee4-b705-f6a7396d75de-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.904992 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.904811    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.905083 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.904853    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.906639 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.906610    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.906797 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.906782    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.927112 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.927086    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:57:59.927490 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:57:59.927469    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9rbr\" (UniqueName: \"kubernetes.io/projected/2913dd75-7d51-4ee4-b705-f6a7396d75de-kube-api-access-c9rbr\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:58:00.040777 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:00.040699    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:58:00.392432 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:00.392405    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"]
Oct 29 21:58:00.394127 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:58:00.394103    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2913dd75_7d51_4ee4_b705_f6a7396d75de.slice/crio-e4dff8ec1434625f0ff1a675088113513e6b62dc2061b3047686a70ae9f8f233 WatchSource:0}: Error finding container e4dff8ec1434625f0ff1a675088113513e6b62dc2061b3047686a70ae9f8f233: Status 404 returned error can't find the container with id e4dff8ec1434625f0ff1a675088113513e6b62dc2061b3047686a70ae9f8f233
Oct 29 21:58:00.396444 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:00.396412    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:58:00.396532 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:00.396474    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:58:00.396532 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:00.396500    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:58:00.411559 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:00.411459    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk" event={"ID":"2913dd75-7d51-4ee4-b705-f6a7396d75de","Type":"ContainerStarted","Data":"e4dff8ec1434625f0ff1a675088113513e6b62dc2061b3047686a70ae9f8f233"}
Oct 29 21:58:00.532936 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:00.532900    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:00.887167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:00.886820    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 21:58:00.887167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:00.886881    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f" podUID="e32c602e-d9fc-4a01-93f1-39391f91653d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 21:58:01.419661 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:01.419626    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk" event={"ID":"2913dd75-7d51-4ee4-b705-f6a7396d75de","Type":"ContainerStarted","Data":"c690d85e84e3d7874dc412f661d0a0128754f47b29e1d323b07d9852b3881237"}
Oct 29 21:58:01.883596 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:01.883554    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 21:58:01.883894 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:01.883862    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f" podUID="e32c602e-d9fc-4a01-93f1-39391f91653d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 21:58:02.041392 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:02.041344    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:58:02.042984 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:02.042958    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 21:58:02.043124 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:02.043004    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk" podUID="2913dd75-7d51-4ee4-b705-f6a7396d75de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 21:58:02.375183 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:02.375134    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:02.895631 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:02.895603    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:58:02.896264 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:02.896156    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:58:02.897088 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:02.897046    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 21:58:02.976128 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:02.976065    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk" podStartSLOduration=3.976045504 podStartE2EDuration="3.976045504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:01.478462902 +0000 UTC m=+697.635588430" watchObservedRunningTime="2026-10-29 21:58:02.976045504 +0000 UTC m=+699.133171002"
Oct 29 21:58:03.041604 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:03.041347    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 21:58:03.041604 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:03.041412    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk" podUID="2913dd75-7d51-4ee4-b705-f6a7396d75de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 21:58:04.044187 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:04.043465    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 21:58:04.044187 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:04.043527    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk" podUID="2913dd75-7d51-4ee4-b705-f6a7396d75de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 21:58:04.301771 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:04.301677    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:05.041872 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:05.041494    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 21:58:05.041872 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:05.041542    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk" podUID="2913dd75-7d51-4ee4-b705-f6a7396d75de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 21:58:06.041029 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:06.040994    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 21:58:06.041461 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:06.041044    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk" podUID="2913dd75-7d51-4ee4-b705-f6a7396d75de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 21:58:06.421776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:06.421728    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:07.041380 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.041348    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 21:58:07.041780 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.041402    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk" podUID="2913dd75-7d51-4ee4-b705-f6a7396d75de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 21:58:07.584494 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.584446    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"]
Oct 29 21:58:07.588822 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.588798    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.603706 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.603675    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a940bc-openshift-default-dockercfg-cl8t8\""
Oct 29 21:58:07.613864 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.613832    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.614019 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.613875    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.614019 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.613979    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xxq2\" (UniqueName: \"kubernetes.io/projected/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-kube-api-access-7xxq2\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.614104 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.614032    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.614104 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.614073    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.614224 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.614172    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.614376 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.614352    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.614442 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.614399    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.614502 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.614463    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.618885 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.618861    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"]
Oct 29 21:58:07.716059 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.716011    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.716265 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.716079    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.716265 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.716180    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.716265 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.716210    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.716265 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.716235    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.716508 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.716267    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xxq2\" (UniqueName: \"kubernetes.io/projected/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-kube-api-access-7xxq2\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.716508 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.716300    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.716508 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.716337    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.716508 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.716416    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.716508 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.716499    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.716787 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.716658    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.716848 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.716824    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.717260 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.717231    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.717642 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.717619    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.719403 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.719382    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.719575 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.719558    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.741276 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.741244    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.741581 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.741550    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xxq2\" (UniqueName: \"kubernetes.io/projected/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-kube-api-access-7xxq2\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:07.904803 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:07.904720    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:08.220779 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:08.220672    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:08.245390 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:08.245361    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"]
Oct 29 21:58:08.246949 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:58:08.246920    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5058fbcb_91cf_44bd_9d4c_34f2acc83f23.slice/crio-ba7935c5473e6fd9a179568fb2169eb14d15544c5efb16ff8071248001c858fa WatchSource:0}: Error finding container ba7935c5473e6fd9a179568fb2169eb14d15544c5efb16ff8071248001c858fa: Status 404 returned error can't find the container with id ba7935c5473e6fd9a179568fb2169eb14d15544c5efb16ff8071248001c858fa
Oct 29 21:58:08.250763 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:08.250026    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:58:08.250763 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:08.250138    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:58:08.250763 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:08.250209    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:58:08.470733 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:08.467940    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" event={"ID":"5058fbcb-91cf-44bd-9d4c-34f2acc83f23","Type":"ContainerStarted","Data":"7d7ca906a936c6886783427317e3064c6881c3da32b11a21f526989c1999b7b2"}
Oct 29 21:58:08.470733 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:08.468008    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" event={"ID":"5058fbcb-91cf-44bd-9d4c-34f2acc83f23","Type":"ContainerStarted","Data":"ba7935c5473e6fd9a179568fb2169eb14d15544c5efb16ff8071248001c858fa"}
Oct 29 21:58:08.513040 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:08.512958    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:58:08.513405 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:08.513372    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:58:08.514350 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:08.514326    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 21:58:08.520117 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:08.520063    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" podStartSLOduration=1.520046241 podStartE2EDuration="1.520046241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:08.516129899 +0000 UTC m=+704.673255452" watchObservedRunningTime="2026-10-29 21:58:08.520046241 +0000 UTC m=+704.677171739"
Oct 29 21:58:08.905011 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:08.904975    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:09.907224 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:09.907186    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 21:58:09.907706 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:09.907238    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 21:58:09.913965 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:09.913933    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:10.861664 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:10.861625    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:10.906040 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:10.906005    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 21:58:10.906224 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:10.906055    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 21:58:11.507894 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:58:11.507859    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:58:11.647958 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:11.647921    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:11.905021 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:11.904987    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 21:58:11.905256 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:11.905050    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 21:58:12.905127 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:12.905097    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 21:58:12.905543 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:12.905162    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 21:58:13.343717 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:13.343672    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:13.905206 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:13.905171    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 21:58:13.905708 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:13.905223    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 21:58:14.905576 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:14.905544    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 29 21:58:14.905963 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:14.905591    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 29 21:58:15.048737 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:15.048695    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:15.910823 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:15.910792    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:15.911314 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:15.911265    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:15.912037 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:15.912019    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 21:58:16.650360 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:16.650319    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:18.269443 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:18.269404    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:18.793608 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:18.793574    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:19.923364 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:19.923326    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:21.312286 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:21.312239    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:21.476974 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:21.476931    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:23.110039 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:23.109970    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:24.675763 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:24.675725    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:26.294528 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:26.294468    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:28.004840 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:28.004802    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:29.601257 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:29.601215    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:31.398266 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.398219    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:31.689937 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.689838    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"]
Oct 29 21:58:31.698039 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.698002    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.728307 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.728277    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"]
Oct 29 21:58:31.813919 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.813888    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.814094 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.813934    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.814094 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.813962    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.814094 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.814036    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.814223 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.814167    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.814223 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.814208    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.814286 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.814257    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.814336 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.814323    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gppr\" (UniqueName: \"kubernetes.io/projected/2fbcc15a-7f6f-4742-aabb-916bb609889a-kube-api-access-5gppr\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.814413 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.814357    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fbcc15a-7f6f-4742-aabb-916bb609889a-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.915514 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.915472    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gppr\" (UniqueName: \"kubernetes.io/projected/2fbcc15a-7f6f-4742-aabb-916bb609889a-kube-api-access-5gppr\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.915703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.915528    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fbcc15a-7f6f-4742-aabb-916bb609889a-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.915703 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.915603    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.915824 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.915738    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.915824 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.915800    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.915925 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.915839    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.915925 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.915917    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.916022 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.915958    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.916022 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.915990    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.916323 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.916298    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.916503 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.916299    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.916638 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.916372    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fbcc15a-7f6f-4742-aabb-916bb609889a-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.916638 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.916402    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.916812 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.916775    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.918066 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.918042    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.918901 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.918870    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.940256 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.940184    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:31.940621 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:31.940604    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gppr\" (UniqueName: \"kubernetes.io/projected/2fbcc15a-7f6f-4742-aabb-916bb609889a-kube-api-access-5gppr\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:32.014031 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:32.013998    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:32.338608 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:32.338568    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"]
Oct 29 21:58:32.352237 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:58:32.352205    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbcc15a_7f6f_4742_aabb_916bb609889a.slice/crio-d46cc86a5bcc28f44f514a171e8f30d83a7006aaec31226e1907f415a000a83a WatchSource:0}: Error finding container d46cc86a5bcc28f44f514a171e8f30d83a7006aaec31226e1907f415a000a83a: Status 404 returned error can't find the container with id d46cc86a5bcc28f44f514a171e8f30d83a7006aaec31226e1907f415a000a83a
Oct 29 21:58:32.355378 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:32.355331    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:58:32.355491 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:32.355418    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:58:32.355491 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:32.355454    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:58:32.641872 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:32.641782    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628" event={"ID":"2fbcc15a-7f6f-4742-aabb-916bb609889a","Type":"ContainerStarted","Data":"9f4e64e0dc5a927dafc93db5a58cfdd5c7d60552a85702a8b060f93e201b316d"}
Oct 29 21:58:32.641872 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:32.641844    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628" event={"ID":"2fbcc15a-7f6f-4742-aabb-916bb609889a","Type":"ContainerStarted","Data":"d46cc86a5bcc28f44f514a171e8f30d83a7006aaec31226e1907f415a000a83a"}
Oct 29 21:58:33.016406 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:33.014935    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:33.017666 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:33.017537    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 29 21:58:33.017666 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:33.017596    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628" podUID="2fbcc15a-7f6f-4742-aabb-916bb609889a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 29 21:58:33.185165 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:33.185068    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:34.014398 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:34.014364    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 29 21:58:34.014981 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:34.014417    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628" podUID="2fbcc15a-7f6f-4742-aabb-916bb609889a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 29 21:58:35.013632 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:35.013601    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:35.014637 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:35.014594    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 29 21:58:35.014989 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:35.014658    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628" podUID="2fbcc15a-7f6f-4742-aabb-916bb609889a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 29 21:58:36.015224 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:36.015186    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 29 21:58:36.015609 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:36.015246    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628" podUID="2fbcc15a-7f6f-4742-aabb-916bb609889a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 29 21:58:36.712676 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:36.712632    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:37.015086 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:37.015006    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 29 21:58:37.015086 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:37.015056    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628" podUID="2fbcc15a-7f6f-4742-aabb-916bb609889a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 29 21:58:37.840534 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:37.840498    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:38.014682 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:38.014650    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 29 21:58:38.014859 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:38.014704    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628" podUID="2fbcc15a-7f6f-4742-aabb-916bb609889a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 29 21:58:38.539253 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:38.539214    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:38.667776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:38.667727    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:39.015119 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:39.015080    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 29 21:58:39.015314 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:39.015167    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628" podUID="2fbcc15a-7f6f-4742-aabb-916bb609889a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 29 21:58:40.019732 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:40.019700    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:40.020235 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:40.020051    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:40.020773 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:40.020750    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 21:58:40.079251 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:40.079207    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628" podStartSLOduration=9.079195497 podStartE2EDuration="9.079195497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:32.700280318 +0000 UTC m=+728.857405817" watchObservedRunningTime="2026-10-29 21:58:40.079195497 +0000 UTC m=+736.236320994"
Oct 29 21:58:40.217934 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:40.217898    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:41.490802 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:58:41.490766    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:58:41.920130 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:41.920084    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:43.773459 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:43.773423    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:45.336103 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:45.336071    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:47.034774 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:47.034740    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:48.622116 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:48.622081    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:48.940254 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:48.940169    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"]
Oct 29 21:58:48.944978 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:48.944957    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:48.961181 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:48.961158    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a9def7-openshift-default-dockercfg-qflmj\""
Oct 29 21:58:48.975331 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:48.975303    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"]
Oct 29 21:58:49.033815 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.033778    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.033993 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.033820    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.033993 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.033870    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.033993 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.033916    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.034098 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.034012    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.034098 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.034032    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.034098 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.034074    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-878dg\" (UniqueName: \"kubernetes.io/projected/7fe0d723-a137-4c93-9657-ca9258b08a27-kube-api-access-878dg\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.034229 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.034218    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe0d723-a137-4c93-9657-ca9258b08a27-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.034262 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.034250    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.135627 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.135589    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-878dg\" (UniqueName: \"kubernetes.io/projected/7fe0d723-a137-4c93-9657-ca9258b08a27-kube-api-access-878dg\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.135837 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.135721    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe0d723-a137-4c93-9657-ca9258b08a27-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.135837 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.135761    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.135971 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.135935    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.136034 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.135974    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.136093 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.136039    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.136181 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.136095    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.136254 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.136206    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.136254 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.136236    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.136362 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.136345    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.136606 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.136582    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.136842 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.136822    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.137008 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.136982    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe0d723-a137-4c93-9657-ca9258b08a27-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.137123 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.137091    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.139072 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.139045    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.139451 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.139429    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.158729 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.158707    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-878dg\" (UniqueName: \"kubernetes.io/projected/7fe0d723-a137-4c93-9657-ca9258b08a27-kube-api-access-878dg\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.158819 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.158752    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.259805 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.259720    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:49.585227 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:58:49.585137    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe0d723_a137_4c93_9657_ca9258b08a27.slice/crio-d49e3aadecfb96a2d1ef7029ed419f05f7ed9e821b1e92c0d199d08850facac7 WatchSource:0}: Error finding container d49e3aadecfb96a2d1ef7029ed419f05f7ed9e821b1e92c0d199d08850facac7: Status 404 returned error can't find the container with id d49e3aadecfb96a2d1ef7029ed419f05f7ed9e821b1e92c0d199d08850facac7
Oct 29 21:58:49.585227 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.585206    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"]
Oct 29 21:58:49.587619 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.587587    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:58:49.587692 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.587655    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:58:49.587692 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.587684    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:58:49.785167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.785112    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp" event={"ID":"7fe0d723-a137-4c93-9657-ca9258b08a27","Type":"ContainerStarted","Data":"34d3593ee7810d93590353aefc91ba1dad61c15c9a3a7b49ec8514e771475898"}
Oct 29 21:58:49.785639 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.785183    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp" event={"ID":"7fe0d723-a137-4c93-9657-ca9258b08a27","Type":"ContainerStarted","Data":"d49e3aadecfb96a2d1ef7029ed419f05f7ed9e821b1e92c0d199d08850facac7"}
Oct 29 21:58:49.833858 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:49.833764    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp" podStartSLOduration=1.8337500759999998 podStartE2EDuration="1.833750076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:49.833001776 +0000 UTC m=+745.990127278" watchObservedRunningTime="2026-10-29 21:58:49.833750076 +0000 UTC m=+745.990875644"
Oct 29 21:58:50.260420 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:50.260335    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:50.274165 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:50.266851    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 29 21:58:50.274165 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:50.266908    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp" podUID="7fe0d723-a137-4c93-9657-ca9258b08a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 29 21:58:50.331117 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:50.331084    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:50.557117 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:50.557023    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:51.260890 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:51.260859    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 29 21:58:51.261339 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:51.260916    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp" podUID="7fe0d723-a137-4c93-9657-ca9258b08a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 29 21:58:52.043659 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:52.043626    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:52.260302 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:52.260269    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 29 21:58:52.260474 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:52.260318    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp" podUID="7fe0d723-a137-4c93-9657-ca9258b08a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 29 21:58:53.260443 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:53.260407    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 29 21:58:53.260872 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:53.260468    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp" podUID="7fe0d723-a137-4c93-9657-ca9258b08a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 29 21:58:53.636838 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:53.636798    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:53.885171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:53.881368    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"]
Oct 29 21:58:53.891645 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:53.891583    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:53.909812 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:53.909787    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8a9e5f-openshift-default-dockercfg-98j45\""
Oct 29 21:58:53.922486 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:53.922461    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"]
Oct 29 21:58:54.014390 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.014354    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.014553 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.014399    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.014553 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.014419    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqttt\" (UniqueName: \"kubernetes.io/projected/bdcde015-efb5-4689-84f3-6e65620b8a53-kube-api-access-kqttt\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.014553 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.014539    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.014664 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.014622    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.014762 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.014746    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.014797 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.014787    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.014833 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.014813    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdcde015-efb5-4689-84f3-6e65620b8a53-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.014925 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.014911    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.115859 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.115822    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.116036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.115873    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.116036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.115889    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqttt\" (UniqueName: \"kubernetes.io/projected/bdcde015-efb5-4689-84f3-6e65620b8a53-kube-api-access-kqttt\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.116036 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.116000    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.116278 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.116061    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.116278 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.116161    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.116278 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.116182    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.116278 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.116197    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdcde015-efb5-4689-84f3-6e65620b8a53-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.116278 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.116258    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.116553 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.116284    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.116553 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.116365    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.116553 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.116385    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.116724 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.116628    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.116941 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.116923    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdcde015-efb5-4689-84f3-6e65620b8a53-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.118754 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.118724    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.118926 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.118910    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.139813 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.139787    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.139927 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.139913    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqttt\" (UniqueName: \"kubernetes.io/projected/bdcde015-efb5-4689-84f3-6e65620b8a53-kube-api-access-kqttt\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.206018 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.205949    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:54.292505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.261049    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 29 21:58:54.292505 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.261116    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp" podUID="7fe0d723-a137-4c93-9657-ca9258b08a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 29 21:58:54.531109 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.531078    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"]
Oct 29 21:58:54.532698 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:58:54.532664    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdcde015_efb5_4689_84f3_6e65620b8a53.slice/crio-d99eb7e9dd6416999894271522a8f2e7a121cdcb69e1095659140dec855a6ffe WatchSource:0}: Error finding container d99eb7e9dd6416999894271522a8f2e7a121cdcb69e1095659140dec855a6ffe: Status 404 returned error can't find the container with id d99eb7e9dd6416999894271522a8f2e7a121cdcb69e1095659140dec855a6ffe
Oct 29 21:58:54.535701 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.535671    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:58:54.535789 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.535735    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:58:54.535789 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.535765    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:58:54.836990 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.836951    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" event={"ID":"bdcde015-efb5-4689-84f3-6e65620b8a53","Type":"ContainerStarted","Data":"73823e5c9354fa4d74bf0c904d8de9ebf5d3932aeb8063e28f5f1fb38703bcfb"}
Oct 29 21:58:54.837210 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.837018    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" event={"ID":"bdcde015-efb5-4689-84f3-6e65620b8a53","Type":"ContainerStarted","Data":"d99eb7e9dd6416999894271522a8f2e7a121cdcb69e1095659140dec855a6ffe"}
Oct 29 21:58:54.887168 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:54.885911    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" podStartSLOduration=1.885891886 podStartE2EDuration="1.885891886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:54.884651441 +0000 UTC m=+751.041776969" watchObservedRunningTime="2026-10-29 21:58:54.885891886 +0000 UTC m=+751.043017386"
Oct 29 21:58:55.206747 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:55.206674    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:58:55.208418 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:55.208395    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 21:58:55.208620 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:55.208598    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 21:58:55.260734 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:55.260696    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 29 21:58:55.260876 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:55.260762    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp" podUID="7fe0d723-a137-4c93-9657-ca9258b08a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 29 21:58:55.521229 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:55.517047    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:56.206465 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:56.206418    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 21:58:56.206682 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:56.206475    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 21:58:56.612558 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:56.612512    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:56.613799 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:56.613775    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:56.614747 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:56.614725    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 21:58:57.187410 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:57.187368    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:57.206923 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:57.206888    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 21:58:57.207107 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:57.206969    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 21:58:58.207237 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:58.207207    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 21:58:58.207689 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:58.207664    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 21:58:59.098776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:59.098738    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:58:59.207312 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:59.207269    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 21:58:59.207849 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:58:59.207822    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 21:59:00.115475 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.115432    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"]
Oct 29 21:59:00.123535 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.123505    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.168167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.143619    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"]
Oct 29 21:59:00.202170 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.169106    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.284731 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.284701    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55zwm\" (UniqueName: \"kubernetes.io/projected/be3e55fd-83b9-4c52-a2a9-28da7d04477e-kube-api-access-55zwm\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.311167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.277228    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 21:59:00.311167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.285342    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 21:59:00.311167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.285384    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.311167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.285422    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.311167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.285554    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.311167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.285580    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.311167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.285620    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.311167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.285664    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.311167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.285703    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.410226 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.396012    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.410488 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.410458    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.415357 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.415323    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.452325 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.452266    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.464418 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.464382    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.546707 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.546678    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.570093 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.570058    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.571221 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.571196    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.571414 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.571395    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55zwm\" (UniqueName: \"kubernetes.io/projected/be3e55fd-83b9-4c52-a2a9-28da7d04477e-kube-api-access-55zwm\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.571618 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.521244    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:00.571986 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.440380    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.589192 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.546594    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.589192 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.445800    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.589192 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.550055    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.589192 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.400703    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.589192 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.571073    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.589192 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.506047    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:00.589192 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.531369    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.617384 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.617348    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55zwm\" (UniqueName: \"kubernetes.io/projected/be3e55fd-83b9-4c52-a2a9-28da7d04477e-kube-api-access-55zwm\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.618083 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.618056    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:00.927642 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:00.927608    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:01.072165 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.070982    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:01.234219 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.234106    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 21:59:01.234219 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.234185    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 21:59:01.528477 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.528388    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:01.843372 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.843340    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"]
Oct 29 21:59:01.849660 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.849632    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"]
Oct 29 21:59:01.856480 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.856450    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:01.895234 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.895193    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:01.895378 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.895285    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:01.895378 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.895330    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:01.898453 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.898420    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"]
Oct 29 21:59:01.964340 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.957304    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:01.964340 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.957355    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:01.964340 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.957415    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:01.964340 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.957467    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:01.964340 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.957519    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df7dk\" (UniqueName: \"kubernetes.io/projected/df402fae-12c4-4ed9-9fb1-226a51f66c18-kube-api-access-df7dk\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:01.964340 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.957587    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:01.964340 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.957614    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:01.964340 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.957664    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df402fae-12c4-4ed9-9fb1-226a51f66c18-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:01.964340 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.957733    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:01.964340 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.963944    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 21:59:01.964340 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:01.964011    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:02.013165 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.011006    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h" event={"ID":"be3e55fd-83b9-4c52-a2a9-28da7d04477e","Type":"ContainerStarted","Data":"1ee2f4b4f7b5092a3b10677ad97be15367861bb93fc793ca08cbb1129e01466f"}
Oct 29 21:59:02.058590 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.058428    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.058590 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.058489    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.058590 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.058530    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df7dk\" (UniqueName: \"kubernetes.io/projected/df402fae-12c4-4ed9-9fb1-226a51f66c18-kube-api-access-df7dk\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.058590 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.058574    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.058590 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.058588    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.058911 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.058618    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df402fae-12c4-4ed9-9fb1-226a51f66c18-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.058911 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.058657    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.058911 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.058709    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.058911 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.058724    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.059623 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.059601    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.059991 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.059969    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.060072 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.060023    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.060508 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.060486    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df402fae-12c4-4ed9-9fb1-226a51f66c18-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.060705 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.060689    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.089256 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.071353    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.110193 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.108790    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.111030 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.110972    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.115793 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.114783    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df7dk\" (UniqueName: \"kubernetes.io/projected/df402fae-12c4-4ed9-9fb1-226a51f66c18-kube-api-access-df7dk\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.206452 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.206414    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 21:59:02.207195 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.206843    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 21:59:02.299897 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.299847    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:02.663244 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.654009    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"]
Oct 29 21:59:02.663244 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:59:02.661360    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf402fae_12c4_4ed9_9fb1_226a51f66c18.slice/crio-b568132abc1a2691f14d6407b41c52c3f1444d8229568072026216ffe9344d3d WatchSource:0}: Error finding container b568132abc1a2691f14d6407b41c52c3f1444d8229568072026216ffe9344d3d: Status 404 returned error can't find the container with id b568132abc1a2691f14d6407b41c52c3f1444d8229568072026216ffe9344d3d
Oct 29 21:59:02.664032 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.663511    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:02.664032 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.663618    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:02.664032 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:02.663656    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:03.040554 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:03.033976    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw" event={"ID":"df402fae-12c4-4ed9-9fb1-226a51f66c18","Type":"ContainerStarted","Data":"a14c5909e303534e631514d330176e6344fc2cb135c2feb6c3b192c6bb79af6f"}
Oct 29 21:59:03.040554 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:03.034048    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw" event={"ID":"df402fae-12c4-4ed9-9fb1-226a51f66c18","Type":"ContainerStarted","Data":"b568132abc1a2691f14d6407b41c52c3f1444d8229568072026216ffe9344d3d"}
Oct 29 21:59:03.040554 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:03.037847    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h" event={"ID":"be3e55fd-83b9-4c52-a2a9-28da7d04477e","Type":"ContainerStarted","Data":"1dfd9bcf245f00955db5eb0a1fb055e5845e44aee42dc1e805567c8e4ebb0956"}
Oct 29 21:59:03.130163 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:03.129855    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw" podStartSLOduration=2.129840205 podStartE2EDuration="2.129840205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:03.115954296 +0000 UTC m=+759.273079798" watchObservedRunningTime="2026-10-29 21:59:03.129840205 +0000 UTC m=+759.286965702"
Oct 29 21:59:03.209169 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:03.207422    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h" podStartSLOduration=3.207400826 podStartE2EDuration="3.207400826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:03.185121515 +0000 UTC m=+759.342247007" watchObservedRunningTime="2026-10-29 21:59:03.207400826 +0000 UTC m=+759.364526325"
Oct 29 21:59:03.301067 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:03.300306    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:03.302121 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:03.302093    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 29 21:59:03.302240 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:03.302172    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw" podUID="df402fae-12c4-4ed9-9fb1-226a51f66c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 29 21:59:03.526511 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:03.526476    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:03.931383 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:03.931300    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:03.938379 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:03.938344    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 21:59:03.938628 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:03.938600    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h" podUID="be3e55fd-83b9-4c52-a2a9-28da7d04477e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 21:59:04.251509 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:04.251395    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 21:59:04.251509 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:04.251476    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:04.303167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:04.302551    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 29 21:59:04.303167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:04.302612    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw" podUID="df402fae-12c4-4ed9-9fb1-226a51f66c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 29 21:59:04.529078 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:04.528983    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:59:04.529514 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:04.529493    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:59:04.531170 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:04.530875    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 21:59:04.928673 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:04.928640    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 21:59:04.928841 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:04.928690    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h" podUID="be3e55fd-83b9-4c52-a2a9-28da7d04477e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 21:59:05.140539 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.140496    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"]
Oct 29 21:59:05.144860 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.144834    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.157137 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.157110    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.157260 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.157199    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.157313 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.157249    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.157313 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.157294    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnrs2\" (UniqueName: \"kubernetes.io/projected/db332eb5-5ced-43e9-953b-5d94aa37d706-kube-api-access-hnrs2\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.157405 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.157314    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db332eb5-5ced-43e9-953b-5d94aa37d706-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.157484 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.157466    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.157522 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.157492    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.157656 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.157636    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.157730 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.157712    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.183447 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.183375    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-743a5e-openshift-default-dockercfg-rnvpp\""
Oct 29 21:59:05.216881 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.216846    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"]
Oct 29 21:59:05.258965 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.258932    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnrs2\" (UniqueName: \"kubernetes.io/projected/db332eb5-5ced-43e9-953b-5d94aa37d706-kube-api-access-hnrs2\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.260030 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.260010    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db332eb5-5ced-43e9-953b-5d94aa37d706-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.260280 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.260266    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db332eb5-5ced-43e9-953b-5d94aa37d706-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.260478 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.260468    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.260601 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.260590    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.290983 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.260817    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.291232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.289886    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.321180 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.319311    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.321180 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.319416    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.321180 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.319451    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.321180 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.319511    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.321180 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.319543    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.321180 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.320722    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 29 21:59:05.321180 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.320764    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw" podUID="df402fae-12c4-4ed9-9fb1-226a51f66c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 29 21:59:05.321678 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.321486    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.322305 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.322127    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnrs2\" (UniqueName: \"kubernetes.io/projected/db332eb5-5ced-43e9-953b-5d94aa37d706-kube-api-access-hnrs2\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.322935 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.322390    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.322935 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.322416    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.322935 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.322700    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.348114 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.348092    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.411685 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.411631    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:05.459265 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.458419    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:05.915315 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.913756    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"]
Oct 29 21:59:05.917400 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.917365    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:05.917513 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.917429    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:05.917513 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.917457    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:05.928137 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.928114    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 21:59:05.928250 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:05.928178    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h" podUID="be3e55fd-83b9-4c52-a2a9-28da7d04477e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 21:59:06.059159 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:06.058977    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf" event={"ID":"db332eb5-5ced-43e9-953b-5d94aa37d706","Type":"ContainerStarted","Data":"6ca09127de40590aa8ab387ad1b33780deb7369993fd708742ee536221a1281f"}
Oct 29 21:59:06.059159 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:06.059028    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf" event={"ID":"db332eb5-5ced-43e9-953b-5d94aa37d706","Type":"ContainerStarted","Data":"cdeff4dd051f02ec6b052a4092c8b18c8c25e6b83898594b66595df20f67d820"}
Oct 29 21:59:06.123403 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:06.121665    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf" podStartSLOduration=1.121643624 podStartE2EDuration="1.121643624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:06.120646791 +0000 UTC m=+762.277772289" watchObservedRunningTime="2026-10-29 21:59:06.121643624 +0000 UTC m=+762.278769123"
Oct 29 21:59:06.308100 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:06.307570    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 29 21:59:06.308100 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:06.307638    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw" podUID="df402fae-12c4-4ed9-9fb1-226a51f66c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 29 21:59:06.459302 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:06.459227    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:06.930569 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:06.930530    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 21:59:06.930872 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:06.930844    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h" podUID="be3e55fd-83b9-4c52-a2a9-28da7d04477e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 21:59:07.304654 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:07.304556    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 29 21:59:07.304930 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:07.304895    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw" podUID="df402fae-12c4-4ed9-9fb1-226a51f66c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 29 21:59:07.460924 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:07.460891    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 21:59:07.461427 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:07.460942    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf" podUID="db332eb5-5ced-43e9-953b-5d94aa37d706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 21:59:07.552500 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:07.552459    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:07.868163 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:07.864585    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:08.024192 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:08.020611    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 21:59:08.024192 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:08.020710    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h" podUID="be3e55fd-83b9-4c52-a2a9-28da7d04477e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 21:59:08.303743 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:08.303658    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 29 21:59:08.304039 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:08.304002    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw" podUID="df402fae-12c4-4ed9-9fb1-226a51f66c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 29 21:59:08.460167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:08.460094    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 21:59:08.460361 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:08.460175    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf" podUID="db332eb5-5ced-43e9-953b-5d94aa37d706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 21:59:08.929762 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:08.929550    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 21:59:08.929762 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:08.929624    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h" podUID="be3e55fd-83b9-4c52-a2a9-28da7d04477e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 21:59:09.302224 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:09.301559    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 29 21:59:09.302224 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:09.301641    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw" podUID="df402fae-12c4-4ed9-9fb1-226a51f66c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 29 21:59:09.463170 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:09.459588    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 21:59:09.463170 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:09.459699    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf" podUID="db332eb5-5ced-43e9-953b-5d94aa37d706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 21:59:09.804709 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:09.804663    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:10.206163 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.203871    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"]
Oct 29 21:59:10.212499 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.209327    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.235448 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.234673    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"]
Oct 29 21:59:10.243226 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.243200    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"e2e-conflict-b-eab27cf7-kserve-self-signed-certs\""
Oct 29 21:59:10.257022 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.256931    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5249bdad-e5d9-46e1-96e2-643af44555e4-tls-certs\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.264265 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.264231    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c2hv\" (UniqueName: \"kubernetes.io/projected/5249bdad-e5d9-46e1-96e2-643af44555e4-kube-api-access-5c2hv\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.264611 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.264588    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-model-cache\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.264829 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.264810    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-dshm\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.268166 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.265052    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-home\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.268412 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.268389    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-tmp-dir\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.302592 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.302557    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 29 21:59:10.302845 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.302815    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw" podUID="df402fae-12c4-4ed9-9fb1-226a51f66c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 29 21:59:10.313187 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.308027    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:10.370930 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.370895    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5249bdad-e5d9-46e1-96e2-643af44555e4-tls-certs\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.372609 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.372582    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c2hv\" (UniqueName: \"kubernetes.io/projected/5249bdad-e5d9-46e1-96e2-643af44555e4-kube-api-access-5c2hv\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.377171 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.373180    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-model-cache\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.377719 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.377702    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-dshm\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.378305 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.378287    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-home\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.378783 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.378767    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-tmp-dir\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.383350 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.377650    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-model-cache\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.383546 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.379167    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-tmp-dir\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.383708 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.378689    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-home\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.426334 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.425441    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-dshm\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.426334 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.425994    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5249bdad-e5d9-46e1-96e2-643af44555e4-tls-certs\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.453426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.452203    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c2hv\" (UniqueName: \"kubernetes.io/projected/5249bdad-e5d9-46e1-96e2-643af44555e4-kube-api-access-5c2hv\") pod \"e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:10.466211 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.463667    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 21:59:10.466211 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.463737    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf" podUID="db332eb5-5ced-43e9-953b-5d94aa37d706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 21:59:10.498043 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.489165    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:10.498043 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.490582    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:10.498043 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.492120    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 21:59:10.537527 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:10.537483    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:11.093447 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:11.085319    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"]
Oct 29 21:59:11.099814 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:11.099741    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:11.188337 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:11.184132    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz" event={"ID":"5249bdad-e5d9-46e1-96e2-643af44555e4","Type":"ContainerStarted","Data":"a0608365bbf87400bd71cbf91904bd255fde1fb51457caab8999fd58f8978ef5"}
Oct 29 21:59:11.239056 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:11.237107    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:11.444131 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:11.444100    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:11.444776 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:11.444750    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:11.445998 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:11.445953    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 21:59:11.459082 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:11.459056    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 21:59:11.459298 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:11.459270    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf" podUID="db332eb5-5ced-43e9-953b-5d94aa37d706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 21:59:11.506932 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:59:11.506883    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:59:11.676288 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:11.676245    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:12.193402 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:12.193360    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz" event={"ID":"5249bdad-e5d9-46e1-96e2-643af44555e4","Type":"ContainerStarted","Data":"ee1d5f96e0ad20e47a1e4695bb47ef841a137d519ce2421d4d5990c4bd414101"}
Oct 29 21:59:12.238493 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:12.238424    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz" podStartSLOduration=2.238407929 podStartE2EDuration="2.238407929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:12.236517789 +0000 UTC m=+768.393643306" watchObservedRunningTime="2026-10-29 21:59:12.238407929 +0000 UTC m=+768.395533427"
Oct 29 21:59:12.459374 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:12.459293    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 21:59:12.459374 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:12.459354    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf" podUID="db332eb5-5ced-43e9-953b-5d94aa37d706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 21:59:12.538328 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:12.538296    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:13.357044 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:13.357004    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:13.465316 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:13.465269    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:13.465929 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:13.465689    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:13.466592 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:13.466566    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 21:59:13.745817 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:13.745740    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:15.012340 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:15.012310    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:16.612982 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:16.612947    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:18.243482 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:18.243449    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:19.925829 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:19.925795    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:20.538931 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:20.538893    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:20.551910 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:20.551883    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:21.260930 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:21.260900    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 21:59:21.521246 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:21.521133    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:22.839815 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:22.839784    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"]
Oct 29 21:59:22.845038 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:22.845016    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:22.875804 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:22.875774    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"]
Oct 29 21:59:22.996294 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:22.996260    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:22.996471 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:22.996326    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:22.996471 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:22.996414    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:22.996471 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:22.996451    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:22.996637 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:22.996481    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:22.996700 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:22.996618    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:22.996700 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:22.996665    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:22.996816 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:22.996795    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp8ds\" (UniqueName: \"kubernetes.io/projected/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-kube-api-access-zp8ds\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:22.996900 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:22.996888    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.097976 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.097904    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.097976 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.097962    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.098220 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.098000    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.098220 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.098025    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.098220 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.098065    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.098220 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.098103    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.098220 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.098130    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.098564 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.098275    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp8ds\" (UniqueName: \"kubernetes.io/projected/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-kube-api-access-zp8ds\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.098564 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.098400    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.098676 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.098600    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.098676 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.098657    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.098782 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.098747    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.098863 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.098827    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.099051 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.099028    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.100485 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.100461    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.100604 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.100593    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.119730 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.119700    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:23.122117 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.122095    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.122371 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.122346    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp8ds\" (UniqueName: \"kubernetes.io/projected/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-kube-api-access-zp8ds\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.161889 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.160279    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:23.491936 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.491911    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"]
Oct 29 21:59:23.492212 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:59:23.492190    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f97b3ec_8b10_4b46_b8e0_233c3ce8c9c1.slice/crio-57c8355070ec842d63d3376bf1302aa35e781c0dd170e69592e62c4ec86bdc73 WatchSource:0}: Error finding container 57c8355070ec842d63d3376bf1302aa35e781c0dd170e69592e62c4ec86bdc73: Status 404 returned error can't find the container with id 57c8355070ec842d63d3376bf1302aa35e781c0dd170e69592e62c4ec86bdc73
Oct 29 21:59:23.494603 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.494571    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:23.494657 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.494648    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:23.494700 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:23.494688    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:24.292066 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:24.292031    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" event={"ID":"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1","Type":"ContainerStarted","Data":"be88438cf85dfcb626c515343616bbecfba1c8709d0531fdd21a87275daa7d98"}
Oct 29 21:59:24.292581 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:24.292093    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" event={"ID":"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1","Type":"ContainerStarted","Data":"57c8355070ec842d63d3376bf1302aa35e781c0dd170e69592e62c4ec86bdc73"}
Oct 29 21:59:24.389164 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:24.385966    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" podStartSLOduration=2.385940166 podStartE2EDuration="2.385940166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:24.350545406 +0000 UTC m=+780.507670907" watchObservedRunningTime="2026-10-29 21:59:24.385940166 +0000 UTC m=+780.543065664"
Oct 29 21:59:24.848390 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:24.848340    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:25.166221 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:25.165440    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:25.180262 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:25.180234    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 29 21:59:25.180522 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:25.180487    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 29 21:59:26.165576 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.165536    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 29 21:59:26.166216 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.166183    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 29 21:59:26.515264 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.515183    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"]
Oct 29 21:59:26.520186 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.520135    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.547944 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.547916    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-743a5e\"/\"distinct-w5-743a5e-kserve-self-signed-certs\""
Oct 29 21:59:26.550837 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.550810    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"]
Oct 29 21:59:26.552465 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.552439    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:26.645274 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.645234    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-tmp-dir\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.645451 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.645298    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj9p2\" (UniqueName: \"kubernetes.io/projected/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-kube-api-access-xj9p2\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.645451 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.645378    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-tls-certs\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.645532 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.645473    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-model-cache\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.645605 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.645557    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-home\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.645649 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.645626    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-dshm\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.682225 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.682189    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:26.752270 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.752237    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-model-cache\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.752627 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.752607    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-home\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.752798 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.752777    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-dshm\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.753002 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.752986    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-tmp-dir\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.753134 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.753119    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj9p2\" (UniqueName: \"kubernetes.io/projected/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-kube-api-access-xj9p2\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.753355 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.753338    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-tls-certs\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.753551 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.753528    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-home\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.754492 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.754474    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-tmp-dir\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.754638 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.754458    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-model-cache\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.766515 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.766444    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-dshm\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.767554 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.767529    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-tls-certs\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.790856 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.790825    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj9p2\" (UniqueName: \"kubernetes.io/projected/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-kube-api-access-xj9p2\") pod \"distinct-w5-743a5e-kserve-6c6d6c9585-nd29g\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:26.834609 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:26.834577    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:27.185575 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:27.185524    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 29 21:59:27.186079 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:27.185592    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 29 21:59:27.259576 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:27.259547    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"]
Oct 29 21:59:27.311694 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:27.311664    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" event={"ID":"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd","Type":"ContainerStarted","Data":"ff79c1799bd2704d69ba8a0d86c37639fec189c15cee154b01079948f1f27cce"}
Oct 29 21:59:28.182165 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:28.169528    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 29 21:59:28.182165 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:28.169592    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 29 21:59:28.318607 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:28.318549    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" event={"ID":"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd","Type":"ContainerStarted","Data":"f51df323877787bc7c33895e982d70894d2ce0dd8f694ce45547a5d5ebda9993"}
Oct 29 21:59:28.332767 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:28.332715    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:28.380167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:28.376195    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podStartSLOduration=2.376173819 podStartE2EDuration="2.376173819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:28.361372097 +0000 UTC m=+784.518497616" watchObservedRunningTime="2026-10-29 21:59:28.376173819 +0000 UTC m=+784.533299614"
Oct 29 21:59:28.836571 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:28.836535    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:29.181220 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:29.163494    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 29 21:59:29.181220 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:29.163554    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 29 21:59:30.065186 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:30.065128    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:30.162868 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:30.162836    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 29 21:59:30.163077 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:30.162892    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 29 21:59:31.167030 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:31.167000    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:31.167467 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:31.167364    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:31.168232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:31.168212    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 21:59:31.687525 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:31.687494    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:32.244757 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:32.244701    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:33.363365 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:33.363324    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:35.167159 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:35.167085    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:36.260863 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.260826    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"]
Oct 29 21:59:36.342519 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.342480    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:36.468777 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.468744    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"]
Oct 29 21:59:36.470089 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.470064    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.499557 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.499509    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a940bc\"/\"distinct-2-w6-a940bc-kserve-self-signed-certs\""
Oct 29 21:59:36.611959 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.611913    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-tls-certs\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.680336 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.680296    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-model-cache\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.681479 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.681448    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p9kc\" (UniqueName: \"kubernetes.io/projected/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-kube-api-access-4p9kc\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.681775 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.681752    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-tmp-dir\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.689258 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.689230    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-home\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.689530 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.689510    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-dshm\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.838167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.829066    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p9kc\" (UniqueName: \"kubernetes.io/projected/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-kube-api-access-4p9kc\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.838167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.829248    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-tmp-dir\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.838167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.829296    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-home\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.838167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.829860    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-tmp-dir\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.838167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.834360    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-dshm\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.838167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.835059    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-tls-certs\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.841132 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.840293    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:36.846833 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.846809    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-model-cache\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.847040 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.846378    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-model-cache\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.870262 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.870233    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-home\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.881990 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.881627    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-dshm\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.882754 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.882726    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p9kc\" (UniqueName: \"kubernetes.io/projected/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-kube-api-access-4p9kc\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.891818 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.891762    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-tls-certs\") pod \"distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:36.954940 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:36.954615    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:37.027456 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:37.027366    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:37.516639 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:37.516527    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 21:59:38.079232 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:59:38.071245    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20bda0fc_d2ac_4435_9e5d_bc6b557e59b0.slice/crio-0869884d3755931bd130a5e0a878fb66dcaa6e6c6e92b318fc7a2fbcbfe51b91 WatchSource:0}: Error finding container 0869884d3755931bd130a5e0a878fb66dcaa6e6c6e92b318fc7a2fbcbfe51b91: Status 404 returned error can't find the container with id 0869884d3755931bd130a5e0a878fb66dcaa6e6c6e92b318fc7a2fbcbfe51b91
Oct 29 21:59:38.130178 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:38.127103    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:38.288450 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:38.285733    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"]
Oct 29 21:59:38.535439 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:38.534472    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" event={"ID":"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0","Type":"ContainerStarted","Data":"0869884d3755931bd130a5e0a878fb66dcaa6e6c6e92b318fc7a2fbcbfe51b91"}
Oct 29 21:59:39.616163 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:39.593414    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" event={"ID":"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0","Type":"ContainerStarted","Data":"66cbc33972c6cde7b16aa2fe38b8002cae3a26063ddba3f2daf6c8deafaca86c"}
Oct 29 21:59:39.656920 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:39.653226    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podStartSLOduration=3.653205966 podStartE2EDuration="3.653205966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:39.650486486 +0000 UTC m=+795.807612011" watchObservedRunningTime="2026-10-29 21:59:39.653205966 +0000 UTC m=+795.810331468"
Oct 29 21:59:40.047222 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:40.043784    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:40.266168 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:40.259129    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz" podUID="5249bdad-e5d9-46e1-96e2-643af44555e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:40.337345 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:40.337305    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:40.508167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:40.496620    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:40.508167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:40.500351    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:41.036875 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:41.032700    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:41.489929 ip-10-0-128-85 kubenswrapper[4550]: E1029 21:59:41.489888    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 21:59:42.836063 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:42.836028    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:44.533847 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:44.533814    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:46.107977 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:46.107932    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:46.530713 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:46.530602    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:47.028225 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:47.028194    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:47.040929 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:47.040903    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:47.665246 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:47.665217    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 21:59:47.827818 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:47.827787    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:49.508385 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:49.508347    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:51.134801 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:51.134754    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:52.790503 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:52.790464    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:52.842263 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:52.842229    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"]
Oct 29 21:59:53.024168 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.023537    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:53.187432 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.187403    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"]
Oct 29 21:59:53.193043 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.193021    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.225331 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.225300    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"]
Oct 29 21:59:53.262670 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.262643    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.262858 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.262812    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.262858 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.262840    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.262989 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.262863    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49hzn\" (UniqueName: \"kubernetes.io/projected/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-kube-api-access-49hzn\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.262989 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.262920    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.262989 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.262964    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.263161 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.263042    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.263161 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.263080    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.263161 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.263123    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.364714 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.364670    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.364904 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.364741    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.364984 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.364904    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.364984 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.364929    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.364984 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.364953    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49hzn\" (UniqueName: \"kubernetes.io/projected/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-kube-api-access-49hzn\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.365117 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.365007    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.365117 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.365051    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.365232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.365124    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.365232 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.365175    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.365759 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.365730    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.365879 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.365836    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.365951 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.365875    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.365951 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.365929    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.366250 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.366228    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.367629 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.367604    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.367943 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.367924    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.389132 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.389101    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.389382 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.389177    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49hzn\" (UniqueName: \"kubernetes.io/projected/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-kube-api-access-49hzn\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.510408 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.510334    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:53.832093 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.832069    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"]
Oct 29 21:59:53.833671 ip-10-0-128-85 kubenswrapper[4550]: W1029 21:59:53.833641    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0953b3_a4b1_425f_bf8c_1b0dc847b38c.slice/crio-1815cf6c397098e29cd86c9c2f5378971a5c64527b581b31253049c3c7391a70 WatchSource:0}: Error finding container 1815cf6c397098e29cd86c9c2f5378971a5c64527b581b31253049c3c7391a70: Status 404 returned error can't find the container with id 1815cf6c397098e29cd86c9c2f5378971a5c64527b581b31253049c3c7391a70
Oct 29 21:59:53.836658 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.836621    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:53.836742 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.836702    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:53.836787 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:53.836744    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:54.214513 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.214426    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"]
Oct 29 21:59:54.219816 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.219788    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.256196 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.253111    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"]
Oct 29 21:59:54.258328 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.256782    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"]
Oct 29 21:59:54.261339 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.261318    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.290426 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.286206    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"]
Oct 29 21:59:54.378999 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.378970    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.379181 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379007    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.379181 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379025    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.379181 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379041    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.379181 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379171    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26797f1-f34c-44e3-86e9-dd0ec24af003-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.379333 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379264    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.379333 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379320    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.379428 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379352    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69fdc\" (UniqueName: \"kubernetes.io/projected/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-kube-api-access-69fdc\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.379499 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379432    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.379499 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379485    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.379599 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379521    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.379599 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379548    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9rvp\" (UniqueName: \"kubernetes.io/projected/a26797f1-f34c-44e3-86e9-dd0ec24af003-kube-api-access-v9rvp\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.379684 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379652    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.379684 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379676    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.379782 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379700    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.379782 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379753    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.379901 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379883    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.379992 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.379978    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.481583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.481489    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69fdc\" (UniqueName: \"kubernetes.io/projected/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-kube-api-access-69fdc\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.481583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.481532    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.481583 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.481562    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.481889 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.481597    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.481889 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.481622    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9rvp\" (UniqueName: \"kubernetes.io/projected/a26797f1-f34c-44e3-86e9-dd0ec24af003-kube-api-access-v9rvp\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.481889 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.481716    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.481889 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.481747    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.481889 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.481782    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.481889 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.481817    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.481889 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.481875    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.482270 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.481987    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.482270 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.482040    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.482270 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.482095    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.482270 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.482105    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.482270 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.482123    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.482270 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.482199    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.482270 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.482231    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.482270 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.482259    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.482713 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.482291    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26797f1-f34c-44e3-86e9-dd0ec24af003-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.482713 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.482361    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.482713 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.482401    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.482713 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.482425    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.482713 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.482462    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.482713 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.482612    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.482969 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.482786    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.483106 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.483085    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.483310 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.483289    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.484206 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.484175    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26797f1-f34c-44e3-86e9-dd0ec24af003-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.484688 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.484648    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.485046 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.485026    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.485110 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.485092    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.485874 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.485859    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.505927 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.505897    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.506021 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.505990    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.506220 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.506202    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9rvp\" (UniqueName: \"kubernetes.io/projected/a26797f1-f34c-44e3-86e9-dd0ec24af003-kube-api-access-v9rvp\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.506389 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.506369    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69fdc\" (UniqueName: \"kubernetes.io/projected/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-kube-api-access-69fdc\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.534870 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.534849    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:54.592444 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.591768    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:54.699170 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.699109    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" event={"ID":"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c","Type":"ContainerStarted","Data":"e0e901984945f548fc6df40c334a1037b2c6d5d31c857ed14d0ba4d069da0c7a"}
Oct 29 21:59:54.699337 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.699203    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" event={"ID":"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c","Type":"ContainerStarted","Data":"1815cf6c397098e29cd86c9c2f5378971a5c64527b581b31253049c3c7391a70"}
Oct 29 21:59:54.723525 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.723489    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:54.747586 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.747477    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" podStartSLOduration=1.747463373 podStartE2EDuration="1.747463373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:54.745941588 +0000 UTC m=+810.903067107" watchObservedRunningTime="2026-10-29 21:59:54.747463373 +0000 UTC m=+810.904588871"
Oct 29 21:59:54.882038 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.881991    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:54.882522 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.882091    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:54.882522 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.882157    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:54.883484 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.883441    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"]
Oct 29 21:59:54.913411 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.913062    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"]
Oct 29 21:59:54.916059 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.916030    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:54.916182 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.916089    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:54.916182 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:54.916120    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 21:59:55.510614 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:55.510580    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 21:59:55.511971 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:55.511942    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 21:59:55.512054 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:55.511987    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 21:59:55.707526 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:55.707411    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" event={"ID":"a26797f1-f34c-44e3-86e9-dd0ec24af003","Type":"ContainerStarted","Data":"3d9d4df4d210aa83747257be2ea6467fcee5bd0b8644e657a3e9845aa18a4f1f"}
Oct 29 21:59:55.707526 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:55.707483    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" event={"ID":"a26797f1-f34c-44e3-86e9-dd0ec24af003","Type":"ContainerStarted","Data":"e79c96bf1b8e9a2b36a4d6c0ba87df2dcf988489beb1e566c007cd0d963d536d"}
Oct 29 21:59:55.709328 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:55.709299    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" event={"ID":"6ae4f671-bd04-4b77-9e76-17be23e5f5ee","Type":"ContainerStarted","Data":"bc9aa7797542acc8e82b2e5a187d3224b80f8b23c8c3cda73743ff5e9d938841"}
Oct 29 21:59:55.709460 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:55.709344    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" event={"ID":"6ae4f671-bd04-4b77-9e76-17be23e5f5ee","Type":"ContainerStarted","Data":"393ec4c4ee47493e7e29120223402dc90433e9cd94de2be584ca9fe0c2bc19f2"}
Oct 29 21:59:55.798602 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:55.798490    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" podStartSLOduration=1.7984757409999999 podStartE2EDuration="1.798475741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:55.783605205 +0000 UTC m=+811.940730756" watchObservedRunningTime="2026-10-29 21:59:55.798475741 +0000 UTC m=+811.955601238"
Oct 29 21:59:55.832109 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:55.832054    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" podStartSLOduration=1.83204116 podStartE2EDuration="1.83204116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:55.830406891 +0000 UTC m=+811.987532389" watchObservedRunningTime="2026-10-29 21:59:55.83204116 +0000 UTC m=+811.989166657"
Oct 29 21:59:56.516168 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:56.515566    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 21:59:56.516168 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:56.515630    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 21:59:56.539227 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:56.538180    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 21:59:56.539227 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:56.538328    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 29 21:59:56.539227 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:56.538372    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 29 21:59:56.595177 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:56.592572    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 21:59:56.595177 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:56.594463    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 21:59:56.595177 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:56.594520    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" podUID="a26797f1-f34c-44e3-86e9-dd0ec24af003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 21:59:56.690234 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:56.690188    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:57.527168 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:57.526854    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 21:59:57.527168 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:57.526912    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 21:59:57.539166 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:57.537481    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 29 21:59:57.539166 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:57.537545    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 29 21:59:57.598163 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:57.597626    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 21:59:57.598163 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:57.597690    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" podUID="a26797f1-f34c-44e3-86e9-dd0ec24af003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 21:59:57.737168 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:57.732711    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:58.515169 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:58.514191    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 21:59:58.515169 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:58.514258    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 21:59:58.545168 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:58.538581    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 29 21:59:58.545168 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:58.538647    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 29 21:59:58.615166 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:58.612269    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 21:59:58.615166 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:58.612333    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" podUID="a26797f1-f34c-44e3-86e9-dd0ec24af003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 21:59:59.040834 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:59.040792    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 21:59:59.525166 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:59.524571    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 21:59:59.525166 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:59.524636    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 21:59:59.542168 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:59.541305    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 29 21:59:59.542168 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:59.541360    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 29 21:59:59.599167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:59.598247    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 21:59:59.599167 ip-10-0-128-85 kubenswrapper[4550]: I1029 21:59:59.598324    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" podUID="a26797f1-f34c-44e3-86e9-dd0ec24af003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:00:00.325166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.324316    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"]
Oct 29 22:00:00.540167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.533504    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:00:00.540167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.533576    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 22:00:00.546247 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.541256    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:00:00.546247 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.541334    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 29 22:00:00.627217 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.626749    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:00:00.627217 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.626823    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" podUID="a26797f1-f34c-44e3-86e9-dd0ec24af003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:00:00.635206 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.629551    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"]
Oct 29 22:00:00.635206 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.629597    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"]
Oct 29 22:00:00.639346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.638284    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"
Oct 29 22:00:00.643589 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.642935    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"]
Oct 29 22:00:00.643589 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.643040    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.653428 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.653408    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:00:00.720169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.712719    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.720169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.719247    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.720169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.719301    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5507d3ee-40cc-4da6-8ad7-64538d303eaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2\" (UID: \"5507d3ee-40cc-4da6-8ad7-64538d303eaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"
Oct 29 22:00:00.720169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.719410    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.720169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.719446    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.720169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.719493    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.720169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.719518    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.720169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.719573    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9xd2\" (UniqueName: \"kubernetes.io/projected/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-kube-api-access-f9xd2\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.720169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.719600    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.720169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.719633    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.720169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.719752    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fvrd\" (UniqueName: \"kubernetes.io/projected/5507d3ee-40cc-4da6-8ad7-64538d303eaa-kube-api-access-2fvrd\") pod \"maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2\" (UID: \"5507d3ee-40cc-4da6-8ad7-64538d303eaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"
Oct 29 22:00:00.822046 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.821986    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fvrd\" (UniqueName: \"kubernetes.io/projected/5507d3ee-40cc-4da6-8ad7-64538d303eaa-kube-api-access-2fvrd\") pod \"maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2\" (UID: \"5507d3ee-40cc-4da6-8ad7-64538d303eaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"
Oct 29 22:00:00.822296 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.822082    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.822296 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.822109    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.822296 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.822170    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5507d3ee-40cc-4da6-8ad7-64538d303eaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2\" (UID: \"5507d3ee-40cc-4da6-8ad7-64538d303eaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"
Oct 29 22:00:00.822480 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.822295    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.822480 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.822333    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.822480 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.822385    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.822480 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.822412    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.822480 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.822472    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9xd2\" (UniqueName: \"kubernetes.io/projected/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-kube-api-access-f9xd2\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.822761 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.822498    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.822761 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.822532    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.822761 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.822650    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.823237 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.823210    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5507d3ee-40cc-4da6-8ad7-64538d303eaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2\" (UID: \"5507d3ee-40cc-4da6-8ad7-64538d303eaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"
Oct 29 22:00:00.823478 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.823440    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.823570 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.823553    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.823921 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.823905    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.824019 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.823958    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.825559 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.825539    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.825682 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.825661    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.846522 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.846501    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fvrd\" (UniqueName: \"kubernetes.io/projected/5507d3ee-40cc-4da6-8ad7-64538d303eaa-kube-api-access-2fvrd\") pod \"maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2\" (UID: \"5507d3ee-40cc-4da6-8ad7-64538d303eaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"
Oct 29 22:00:00.846609 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.846555    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.847227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.847210    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9xd2\" (UniqueName: \"kubernetes.io/projected/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-kube-api-access-f9xd2\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:00.956001 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:00.955915    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"
Oct 29 22:00:01.016830 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:01.016746    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:01.351457 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:01.351414    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:01.441007 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:01.437435    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"]
Oct 29 22:00:01.539167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:01.523507    4550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:00:01.548291 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:01.545883    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:00:01.548291 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:01.545942    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 29 22:00:01.556031 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:01.555920    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:00:01.556349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:01.556217    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 22:00:01.560181 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:01.560050    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"]
Oct 29 22:00:01.632447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:01.632246    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:00:01.632447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:01.632316    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" podUID="a26797f1-f34c-44e3-86e9-dd0ec24af003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:00:01.634042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:01.634010    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:01.634189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:01.634079    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:01.634189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:01.634106    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:02.098314 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.098276    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2" event={"ID":"5507d3ee-40cc-4da6-8ad7-64538d303eaa","Type":"ContainerStarted","Data":"8f227873ac1f83dd6ac82f5fef1dd736ddddd1acb316cf286b040919c3831385"}
Oct 29 22:00:02.109826 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.109789    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" event={"ID":"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9","Type":"ContainerStarted","Data":"33c3422c262f448be1cfdf14a0d935e658d0728f5914a97e33160f899c39d4a6"}
Oct 29 22:00:02.110070 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.110048    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" event={"ID":"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9","Type":"ContainerStarted","Data":"8b1d0d86b63bd65c44367eebd32fd3a5a010e7ee75d157b99ab263f67da31ce7"}
Oct 29 22:00:02.149809 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.149772    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"]
Oct 29 22:00:02.156629 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.156606    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"
Oct 29 22:00:02.174252 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.174108    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:00:02.212833 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.193187    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"]
Oct 29 22:00:02.212833 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.193890    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" podStartSLOduration=2.193875421 podStartE2EDuration="2.193875421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:02.188352166 +0000 UTC m=+818.345477702" watchObservedRunningTime="2026-10-29 22:00:02.193875421 +0000 UTC m=+818.351000921"
Oct 29 22:00:02.263759 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.263702    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c512471-7b82-40c8-967f-8b96d57ac572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e4503e9-dcnd5\" (UID: \"3c512471-7b82-40c8-967f-8b96d57ac572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"
Oct 29 22:00:02.263951 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.263831    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c512471-7b82-40c8-967f-8b96d57ac572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e4503e9-dcnd5\" (UID: \"3c512471-7b82-40c8-967f-8b96d57ac572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"
Oct 29 22:00:02.264043 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.264024    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjctj\" (UniqueName: \"kubernetes.io/projected/3c512471-7b82-40c8-967f-8b96d57ac572-kube-api-access-gjctj\") pod \"maas-api-revoke-sub-8e4503e9-dcnd5\" (UID: \"3c512471-7b82-40c8-967f-8b96d57ac572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"
Oct 29 22:00:02.366061 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.365965    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjctj\" (UniqueName: \"kubernetes.io/projected/3c512471-7b82-40c8-967f-8b96d57ac572-kube-api-access-gjctj\") pod \"maas-api-revoke-sub-8e4503e9-dcnd5\" (UID: \"3c512471-7b82-40c8-967f-8b96d57ac572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"
Oct 29 22:00:02.366270 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.366080    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c512471-7b82-40c8-967f-8b96d57ac572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e4503e9-dcnd5\" (UID: \"3c512471-7b82-40c8-967f-8b96d57ac572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"
Oct 29 22:00:02.366270 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.366220    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c512471-7b82-40c8-967f-8b96d57ac572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e4503e9-dcnd5\" (UID: \"3c512471-7b82-40c8-967f-8b96d57ac572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"
Oct 29 22:00:02.367179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.366849    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c512471-7b82-40c8-967f-8b96d57ac572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e4503e9-dcnd5\" (UID: \"3c512471-7b82-40c8-967f-8b96d57ac572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"
Oct 29 22:00:02.372460 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.372435    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c512471-7b82-40c8-967f-8b96d57ac572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e4503e9-dcnd5\" (UID: \"3c512471-7b82-40c8-967f-8b96d57ac572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"
Oct 29 22:00:02.389712 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.389687    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjctj\" (UniqueName: \"kubernetes.io/projected/3c512471-7b82-40c8-967f-8b96d57ac572-kube-api-access-gjctj\") pod \"maas-api-revoke-sub-8e4503e9-dcnd5\" (UID: \"3c512471-7b82-40c8-967f-8b96d57ac572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"
Oct 29 22:00:02.491164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.490381    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"
Oct 29 22:00:02.516668 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.516642    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 22:00:02.517076 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.517056    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 22:00:02.517862 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.517842    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 22:00:02.535659 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.535629    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:00:02.535772 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.535689    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 29 22:00:02.605423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.605073    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:00:02.605423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.605130    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" podUID="a26797f1-f34c-44e3-86e9-dd0ec24af003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:00:02.902955 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:02.901942    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"]
Oct 29 22:00:03.017273 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:03.017233    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:03.018795 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:03.018760    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:00:03.018954 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:03.018817    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 29 22:00:03.125497 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:03.123355    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5" event={"ID":"3c512471-7b82-40c8-967f-8b96d57ac572","Type":"ContainerStarted","Data":"3e10d720bfef323194cecbb291b470ee58edcd0bbaa0b36f72ea678330e680b2"}
Oct 29 22:00:03.422164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:03.421132    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:03.596169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:03.591018    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 22:00:03.596169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:03.594832    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 22:00:03.607177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:03.606516    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 22:00:03.646712 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:03.645949    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 22:00:03.647028 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:03.647008    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 22:00:03.649429 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:03.648588    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 22:00:04.017594 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.017556    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:00:04.018088 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.017601    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 29 22:00:04.187648 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.187607    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"]
Oct 29 22:00:04.203061 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.202522    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.213915 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.213882    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"]
Oct 29 22:00:04.348649 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.348619    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.348838 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.348667    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.348838 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.348713    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.348838 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.348759    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.348838 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.348781    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.348838 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.348796    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.349126 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.348910    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.349126 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.349068    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbnxh\" (UniqueName: \"kubernetes.io/projected/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-kube-api-access-cbnxh\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.349253 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.349218    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.451684 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.450759    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.451684 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.450802    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbnxh\" (UniqueName: \"kubernetes.io/projected/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-kube-api-access-cbnxh\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.451684 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.450923    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.451684 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.450989    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.451684 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.451024    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.451684 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.451072    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.451684 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.451159    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.451684 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.451200    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.451684 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.451229    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.453206 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.452655    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.453206 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.453136    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.453532 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.453510    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.453830 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.453750    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.453830 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.453804    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.455055 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.455012    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.455990 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.455969    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.479900 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.479843    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbnxh\" (UniqueName: \"kubernetes.io/projected/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-kube-api-access-cbnxh\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.488974 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.488917    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.556001 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.555968    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:04.941397 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.934714    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"]
Oct 29 22:00:04.941397 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.936319    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:04.941397 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.936444    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:04.941397 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:04.936525    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:05.017688 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.017661    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:00:05.018004 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.017703    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 29 22:00:05.147821 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.147773    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" event={"ID":"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb","Type":"ContainerStarted","Data":"909c1cfa9b74bf779dad64d01d06fc589c01d337c386a8526095d7d53fd69fca"}
Oct 29 22:00:05.147821 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.147829    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" event={"ID":"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb","Type":"ContainerStarted","Data":"c723e071611310a1e078209d9d28e997a10e9969a71c655591514894e19aa5c5"}
Oct 29 22:00:05.201209 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.200138    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" podStartSLOduration=1.200124679 podStartE2EDuration="1.200124679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:05.197198472 +0000 UTC m=+821.354323992" watchObservedRunningTime="2026-10-29 22:00:05.200124679 +0000 UTC m=+821.357250177"
Oct 29 22:00:05.341946 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.341910    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:05.430068 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.429939    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"]
Oct 29 22:00:05.436285 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.436203    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.537194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.530515    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.537194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.530562    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c48198-27e2-4aa6-86e7-389bd62a2666-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.537194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.530581    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.537194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.530926    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp922\" (UniqueName: \"kubernetes.io/projected/37c48198-27e2-4aa6-86e7-389bd62a2666-kube-api-access-mp922\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.537194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.530957    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.537194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.531011    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.537194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.531029    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.537194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.531066    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.537194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.531101    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.538909 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.538883    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"]
Oct 29 22:00:05.569168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.561593    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:05.636674 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.636631    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.636869 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.636696    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c48198-27e2-4aa6-86e7-389bd62a2666-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.636869 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.636724    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.636869 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.636859    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp922\" (UniqueName: \"kubernetes.io/projected/37c48198-27e2-4aa6-86e7-389bd62a2666-kube-api-access-mp922\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.637030 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.636888    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.637030 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.636962    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.637030 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.636992    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.637192 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.637040    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.637192 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.637094    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.638150 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.638109    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.638781 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.638754    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c48198-27e2-4aa6-86e7-389bd62a2666-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.638994 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.638978    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.639758 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.639736    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.639884 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.639864    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.641450 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.641376    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.643684 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.643662    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.693848 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.693788    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.694661 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.694637    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp922\" (UniqueName: \"kubernetes.io/projected/37c48198-27e2-4aa6-86e7-389bd62a2666-kube-api-access-mp922\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:05.859714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:05.859681    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:06.023587 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:06.023553    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:00:06.024083 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:06.023603    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 29 22:00:06.320638 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:00:06.320600    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c48198_27e2_4aa6_86e7_389bd62a2666.slice/crio-365110503a5d2658688423db366116272e5b51020be6695734bd88611cae2fee WatchSource:0}: Error finding container 365110503a5d2658688423db366116272e5b51020be6695734bd88611cae2fee: Status 404 returned error can't find the container with id 365110503a5d2658688423db366116272e5b51020be6695734bd88611cae2fee
Oct 29 22:00:06.321494 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:06.321466    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"]
Oct 29 22:00:06.324181 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:06.324129    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:06.324374 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:06.324349    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:06.324529 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:06.324508    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:06.560063 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:06.559830    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:00:06.560063 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:06.559915    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 29 22:00:07.025375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:07.023650    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:00:07.025375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:07.023717    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 29 22:00:07.233680 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:07.233648    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49" event={"ID":"37c48198-27e2-4aa6-86e7-389bd62a2666","Type":"ContainerStarted","Data":"fbfa9b2f6c802c7485d2468e40c67f974deec08499ccd2cd5bfcab64d0f317f4"}
Oct 29 22:00:07.233845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:07.233697    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49" event={"ID":"37c48198-27e2-4aa6-86e7-389bd62a2666","Type":"ContainerStarted","Data":"365110503a5d2658688423db366116272e5b51020be6695734bd88611cae2fee"}
Oct 29 22:00:07.325245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:07.314058    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49" podStartSLOduration=2.314038294 podStartE2EDuration="2.314038294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:07.29965754 +0000 UTC m=+823.456783033" watchObservedRunningTime="2026-10-29 22:00:07.314038294 +0000 UTC m=+823.471163794"
Oct 29 22:00:07.325245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:07.319698    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:07.557471 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:07.557436    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:00:07.557663 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:07.557497    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 29 22:00:07.860342 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:07.860312    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:07.867185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:07.861692    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:00:07.867185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:07.861744    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49" podUID="37c48198-27e2-4aa6-86e7-389bd62a2666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:00:07.922311 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:07.921598    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz" podUID="5249bdad-e5d9-46e1-96e2-643af44555e4" containerName="main" containerID="cri-o://ee1d5f96e0ad20e47a1e4695bb47ef841a137d519ce2421d4d5990c4bd414101" gracePeriod=45
Oct 29 22:00:08.141749 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.131242    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:00:08.141749 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.131316    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 29 22:00:08.183848 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:00:08.167278    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5249bdad_e5d9_46e1_96e2_643af44555e4.slice/crio-conmon-ee1d5f96e0ad20e47a1e4695bb47ef841a137d519ce2421d4d5990c4bd414101.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:00:08.249421 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.249373    4550 generic.go:356] "Generic (PLEG): container finished" podID="5249bdad-e5d9-46e1-96e2-643af44555e4" containerID="ee1d5f96e0ad20e47a1e4695bb47ef841a137d519ce2421d4d5990c4bd414101" exitCode=0
Oct 29 22:00:08.251207 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.250103    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz" podUID="5249bdad-e5d9-46e1-96e2-643af44555e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": dial tcp 10.133.0.64:8000: connect: connection refused"
Oct 29 22:00:08.251564 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.250157    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz" event={"ID":"5249bdad-e5d9-46e1-96e2-643af44555e4","Type":"ContainerDied","Data":"ee1d5f96e0ad20e47a1e4695bb47ef841a137d519ce2421d4d5990c4bd414101"}
Oct 29 22:00:08.560623 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.560588    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:00:08.560815 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.560645    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 29 22:00:08.861943 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.861518    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:00:08.861943 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.861578    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49" podUID="37c48198-27e2-4aa6-86e7-389bd62a2666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:00:08.869178 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.868225    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 22:00:08.961198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.960662    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5249bdad-e5d9-46e1-96e2-643af44555e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/5249bdad-e5d9-46e1-96e2-643af44555e4-tls-certs\") pod \"5249bdad-e5d9-46e1-96e2-643af44555e4\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") "
Oct 29 22:00:08.967861 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.967827    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5249bdad-e5d9-46e1-96e2-643af44555e4-kube-api-access-5c2hv\" (UniqueName: \"kubernetes.io/projected/5249bdad-e5d9-46e1-96e2-643af44555e4-kube-api-access-5c2hv\") pod \"5249bdad-e5d9-46e1-96e2-643af44555e4\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") "
Oct 29 22:00:08.968183 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.968134    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-model-cache\") pod \"5249bdad-e5d9-46e1-96e2-643af44555e4\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") "
Oct 29 22:00:08.973374 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.973355    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-dshm\") pod \"5249bdad-e5d9-46e1-96e2-643af44555e4\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") "
Oct 29 22:00:08.973927 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.973261    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-model-cache" pod "5249bdad-e5d9-46e1-96e2-643af44555e4" (UID: "5249bdad-e5d9-46e1-96e2-643af44555e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:00:08.986213 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.984228    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-home" pod "5249bdad-e5d9-46e1-96e2-643af44555e4" (UID: "5249bdad-e5d9-46e1-96e2-643af44555e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:00:08.986441 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.986402    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-home\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-home\") pod \"5249bdad-e5d9-46e1-96e2-643af44555e4\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") "
Oct 29 22:00:08.986700 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.986682    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-tmp-dir\") pod \"5249bdad-e5d9-46e1-96e2-643af44555e4\" (UID: \"5249bdad-e5d9-46e1-96e2-643af44555e4\") "
Oct 29 22:00:08.995425 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.995297    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-tmp-dir" pod "5249bdad-e5d9-46e1-96e2-643af44555e4" (UID: "5249bdad-e5d9-46e1-96e2-643af44555e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:00:08.996366 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.996344    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-model-cache\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:00:08.996516 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.996504    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-home\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:00:08.996645 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:08.996632    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-tmp-dir\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:00:09.022556 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.020766    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:00:09.022556 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.020836    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 29 22:00:09.056341 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.056051    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5249bdad-e5d9-46e1-96e2-643af44555e4-tls-certs" pod "5249bdad-e5d9-46e1-96e2-643af44555e4" (UID: "5249bdad-e5d9-46e1-96e2-643af44555e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:00:09.056341 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.056093    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5249bdad-e5d9-46e1-96e2-643af44555e4-kube-api-access-5c2hv" pod "5249bdad-e5d9-46e1-96e2-643af44555e4" (UID: "5249bdad-e5d9-46e1-96e2-643af44555e4"). InnerVolumeSpecName "kube-api-access-5c2hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:00:09.057162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.057116    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-dshm" pod "5249bdad-e5d9-46e1-96e2-643af44555e4" (UID: "5249bdad-e5d9-46e1-96e2-643af44555e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:00:09.120193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.107425    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c2hv\" (UniqueName: \"kubernetes.io/projected/5249bdad-e5d9-46e1-96e2-643af44555e4-kube-api-access-5c2hv\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:00:09.120193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.107463    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5249bdad-e5d9-46e1-96e2-643af44555e4-dshm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:00:09.120193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.107477    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5249bdad-e5d9-46e1-96e2-643af44555e4-tls-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:00:09.277167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.274691    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"
Oct 29 22:00:09.286165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.283681    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz" event={"ID":"5249bdad-e5d9-46e1-96e2-643af44555e4","Type":"ContainerDied","Data":"a0608365bbf87400bd71cbf91904bd255fde1fb51457caab8999fd58f8978ef5"}
Oct 29 22:00:09.286165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.283741    4550 scope.go:122] "RemoveContainer" containerID="ee1d5f96e0ad20e47a1e4695bb47ef841a137d519ce2421d4d5990c4bd414101"
Oct 29 22:00:09.439190 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.436224    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"]
Oct 29 22:00:09.440453 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.440422    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-b-eab27cf7-kserve-85fb8446b4-djtkz"]
Oct 29 22:00:09.513245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.512594    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:09.571219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.567591    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:00:09.571219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.567676    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 29 22:00:09.860968 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.860874    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:00:09.861198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:09.860992    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49" podUID="37c48198-27e2-4aa6-86e7-389bd62a2666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:00:10.010177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:10.004423    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5249bdad-e5d9-46e1-96e2-643af44555e4" path="/var/lib/kubelet/pods/5249bdad-e5d9-46e1-96e2-643af44555e4/volumes"
Oct 29 22:00:10.557553 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:10.557517    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:00:10.558059 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:10.557574    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 29 22:00:10.747587 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:10.746706    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:10.748420 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:10.748393    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:10.750078 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:10.750056    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:00:10.860450 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:10.860412    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:00:10.860650 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:10.860476    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49" podUID="37c48198-27e2-4aa6-86e7-389bd62a2666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:00:11.260120 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.260083    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"]
Oct 29 22:00:11.261076 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.261026    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5249bdad-e5d9-46e1-96e2-643af44555e4" containerName="main"
Oct 29 22:00:11.261076 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.261050    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5249bdad-e5d9-46e1-96e2-643af44555e4" containerName="main"
Oct 29 22:00:11.261333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.261299    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5249bdad-e5d9-46e1-96e2-643af44555e4" containerName="main"
Oct 29 22:00:11.353764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.351044    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:11.379179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.366403    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"]
Oct 29 22:00:11.379179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.366543    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.396739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.396708    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a9def7\"/\"distinct-2-w2-a9def7-kserve-self-signed-certs\""
Oct 29 22:00:11.550222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.533911    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:11.617561 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.617523    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-dshm\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.670391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.669786    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-model-cache\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.670391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.669931    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-tls-certs\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.670391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.669996    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-tmp-dir\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.670391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.670158    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpjkk\" (UniqueName: \"kubernetes.io/projected/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-kube-api-access-qpjkk\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.670391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.670276    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-home\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.696381 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.625810    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:00:11.696381 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.670626    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 29 22:00:11.696381 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.692292    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2" event={"ID":"5507d3ee-40cc-4da6-8ad7-64538d303eaa","Type":"ContainerStarted","Data":"285f58b4dc368040a9e7e95611f97a9370bba4e5ab24702dbbe1fa1f3342bedc"}
Oct 29 22:00:11.755406 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.755351    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2" podStartSLOduration=2.558279076 podStartE2EDuration="11.755334247s" totalImagesPullingTime="9.197055171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:11.750433958 +0000 UTC m=+827.907559456" watchObservedRunningTime="2026-10-29 22:00:11.755334247 +0000 UTC m=+827.912459746"
Oct 29 22:00:11.788167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.785287    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-tmp-dir\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.788167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.785420    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpjkk\" (UniqueName: \"kubernetes.io/projected/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-kube-api-access-qpjkk\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.788167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.785504    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-home\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.788167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.785646    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-dshm\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.788167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.785752    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-model-cache\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.788167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.785852    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-tls-certs\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.817279 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.789390    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-tmp-dir\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.818805 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.817788    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-model-cache\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.818983 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.818039    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-home\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.822902 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.821939    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpjkk\" (UniqueName: \"kubernetes.io/projected/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-kube-api-access-qpjkk\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.823604 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.823578    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-tls-certs\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.823700 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.823611    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-dshm\") pod \"distinct-2-w2-a9def7-kserve-68bb48994-cgk5m\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:11.827976 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:00:11.827949    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:00:11.860455 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.860428    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:00:11.860558 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.860486    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49" podUID="37c48198-27e2-4aa6-86e7-389bd62a2666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:00:11.995412 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:11.985500    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:12.016639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:12.016611    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:12.338737 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:12.338709    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"]
Oct 29 22:00:12.340921 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:00:12.340884    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a90acbd_b3d5_40fe_ac0c_60a84b0dd963.slice/crio-e8e945df186390f30967cd3587c5591cf8f384ac595c4ce9ba5b6e930184d062 WatchSource:0}: Error finding container e8e945df186390f30967cd3587c5591cf8f384ac595c4ce9ba5b6e930184d062: Status 404 returned error can't find the container with id e8e945df186390f30967cd3587c5591cf8f384ac595c4ce9ba5b6e930184d062
Oct 29 22:00:12.556619 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:12.556581    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 29 22:00:12.556786 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:12.556643    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 29 22:00:12.705502 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:12.705406    4550 generic.go:356] "Generic (PLEG): container finished" podID="5507d3ee-40cc-4da6-8ad7-64538d303eaa" containerID="285f58b4dc368040a9e7e95611f97a9370bba4e5ab24702dbbe1fa1f3342bedc" exitCode=60
Oct 29 22:00:12.705887 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:12.705506    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2" event={"ID":"5507d3ee-40cc-4da6-8ad7-64538d303eaa","Type":"ContainerDied","Data":"285f58b4dc368040a9e7e95611f97a9370bba4e5ab24702dbbe1fa1f3342bedc"}
Oct 29 22:00:12.705887 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:12.705733    4550 scope.go:122] "RemoveContainer" containerID="285f58b4dc368040a9e7e95611f97a9370bba4e5ab24702dbbe1fa1f3342bedc"
Oct 29 22:00:12.709099 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:12.709043    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" event={"ID":"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963","Type":"ContainerStarted","Data":"82b0d75de4a3f8bc6d99224e95f838db55c3262182841a111f7d2ad02795a501"}
Oct 29 22:00:12.709099 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:12.709081    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" event={"ID":"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963","Type":"ContainerStarted","Data":"e8e945df186390f30967cd3587c5591cf8f384ac595c4ce9ba5b6e930184d062"}
Oct 29 22:00:12.715385 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:12.711534    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5" event={"ID":"3c512471-7b82-40c8-967f-8b96d57ac572","Type":"ContainerStarted","Data":"c147dd074d943c076fee84eb13776e53b31750043d7b6e8c87fcc5c44e9a0a53"}
Oct 29 22:00:12.861106 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:12.860136    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:00:12.861106 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:12.860207    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49" podUID="37c48198-27e2-4aa6-86e7-389bd62a2666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:00:12.901676 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:12.901614    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5" podStartSLOduration=2.154840633 podStartE2EDuration="10.901593857s" totalImagesPullingTime="8.746753224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:12.89992559 +0000 UTC m=+829.057051090" watchObservedRunningTime="2026-10-29 22:00:12.901593857 +0000 UTC m=+829.058719357"
Oct 29 22:00:12.909600 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:12.909276    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podStartSLOduration=1.909261895 podStartE2EDuration="1.909261895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:12.824447191 +0000 UTC m=+828.981572702" watchObservedRunningTime="2026-10-29 22:00:12.909261895 +0000 UTC m=+829.066387394"
Oct 29 22:00:12.994604 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:12.994539    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:13.561009 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:13.560978    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:13.561419 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:13.561392    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:13.562606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:13.562583    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:00:13.723288 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:13.723249    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2" event={"ID":"5507d3ee-40cc-4da6-8ad7-64538d303eaa","Type":"ContainerStarted","Data":"a9a55f6dcac14d27f74219a5cd3bc59762fda164d068e95c442756c27d89397d"}
Oct 29 22:00:13.724952 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:13.724924    4550 generic.go:356] "Generic (PLEG): container finished" podID="3c512471-7b82-40c8-967f-8b96d57ac572" containerID="c147dd074d943c076fee84eb13776e53b31750043d7b6e8c87fcc5c44e9a0a53" exitCode=0
Oct 29 22:00:13.725109 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:13.725005    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5" event={"ID":"3c512471-7b82-40c8-967f-8b96d57ac572","Type":"ContainerDied","Data":"c147dd074d943c076fee84eb13776e53b31750043d7b6e8c87fcc5c44e9a0a53"}
Oct 29 22:00:13.943201 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:13.938702    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:14.079309 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:14.079273    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:14.080398 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:14.080377    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:14.080974 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:14.080954    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:00:14.733807 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:14.733755    4550 generic.go:356] "Generic (PLEG): container finished" podID="5507d3ee-40cc-4da6-8ad7-64538d303eaa" containerID="a9a55f6dcac14d27f74219a5cd3bc59762fda164d068e95c442756c27d89397d" exitCode=60
Oct 29 22:00:14.734288 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:14.733844    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2" event={"ID":"5507d3ee-40cc-4da6-8ad7-64538d303eaa","Type":"ContainerDied","Data":"a9a55f6dcac14d27f74219a5cd3bc59762fda164d068e95c442756c27d89397d"}
Oct 29 22:00:14.734288 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:14.733966    4550 scope.go:122] "RemoveContainer" containerID="285f58b4dc368040a9e7e95611f97a9370bba4e5ab24702dbbe1fa1f3342bedc"
Oct 29 22:00:14.734288 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:14.734192    4550 scope.go:122] "RemoveContainer" containerID="a9a55f6dcac14d27f74219a5cd3bc59762fda164d068e95c442756c27d89397d"
Oct 29 22:00:14.734605 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:00:14.734582    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2_odh-ai-gateway-infra(5507d3ee-40cc-4da6-8ad7-64538d303eaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2" podUID="5507d3ee-40cc-4da6-8ad7-64538d303eaa"
Oct 29 22:00:14.904760 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:14.904736    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"
Oct 29 22:00:15.061082 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:15.061007    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c512471-7b82-40c8-967f-8b96d57ac572-kube-api-access-gjctj\" (UniqueName: \"kubernetes.io/projected/3c512471-7b82-40c8-967f-8b96d57ac572-kube-api-access-gjctj\") pod \"3c512471-7b82-40c8-967f-8b96d57ac572\" (UID: \"3c512471-7b82-40c8-967f-8b96d57ac572\") "
Oct 29 22:00:15.061292 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:15.061122    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c512471-7b82-40c8-967f-8b96d57ac572-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c512471-7b82-40c8-967f-8b96d57ac572-maas-api-xks-ca\") pod \"3c512471-7b82-40c8-967f-8b96d57ac572\" (UID: \"3c512471-7b82-40c8-967f-8b96d57ac572\") "
Oct 29 22:00:15.061292 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:15.061190    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c512471-7b82-40c8-967f-8b96d57ac572-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c512471-7b82-40c8-967f-8b96d57ac572-maas-api-openshift-ca\") pod \"3c512471-7b82-40c8-967f-8b96d57ac572\" (UID: \"3c512471-7b82-40c8-967f-8b96d57ac572\") "
Oct 29 22:00:15.061625 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:15.061592    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c512471-7b82-40c8-967f-8b96d57ac572-maas-api-openshift-ca" pod "3c512471-7b82-40c8-967f-8b96d57ac572" (UID: "3c512471-7b82-40c8-967f-8b96d57ac572"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:00:15.061750 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:15.061736    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c512471-7b82-40c8-967f-8b96d57ac572-maas-api-openshift-ca\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:00:15.063059 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:15.063035    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c512471-7b82-40c8-967f-8b96d57ac572-kube-api-access-gjctj" pod "3c512471-7b82-40c8-967f-8b96d57ac572" (UID: "3c512471-7b82-40c8-967f-8b96d57ac572"). InnerVolumeSpecName "kube-api-access-gjctj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:00:15.063181 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:15.063100    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c512471-7b82-40c8-967f-8b96d57ac572-maas-api-xks-ca" pod "3c512471-7b82-40c8-967f-8b96d57ac572" (UID: "3c512471-7b82-40c8-967f-8b96d57ac572"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:00:15.163351 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:15.163326    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c512471-7b82-40c8-967f-8b96d57ac572-maas-api-xks-ca\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:00:15.163351 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:15.163349    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjctj\" (UniqueName: \"kubernetes.io/projected/3c512471-7b82-40c8-967f-8b96d57ac572-kube-api-access-gjctj\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:00:15.608357 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:15.608324    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:15.742539 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:15.742512    4550 scope.go:122] "RemoveContainer" containerID="a9a55f6dcac14d27f74219a5cd3bc59762fda164d068e95c442756c27d89397d"
Oct 29 22:00:15.742965 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:00:15.742881    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2_odh-ai-gateway-infra(5507d3ee-40cc-4da6-8ad7-64538d303eaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2" podUID="5507d3ee-40cc-4da6-8ad7-64538d303eaa"
Oct 29 22:00:15.743554 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:15.743534    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"
Oct 29 22:00:15.743650 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:15.743537    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5" event={"ID":"3c512471-7b82-40c8-967f-8b96d57ac572","Type":"ContainerDied","Data":"3e10d720bfef323194cecbb291b470ee58edcd0bbaa0b36f72ea678330e680b2"}
Oct 29 22:00:15.743650 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:15.743599    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e10d720bfef323194cecbb291b470ee58edcd0bbaa0b36f72ea678330e680b2"
Oct 29 22:00:17.316731 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:17.316691    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:19.266166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.265260    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"]
Oct 29 22:00:19.266659 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.266242    4550 cpu_manager.go:414] "containerMap: removing container" podUID="3c512471-7b82-40c8-967f-8b96d57ac572" containerName="revoke-keys"
Oct 29 22:00:19.266659 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.266265    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c512471-7b82-40c8-967f-8b96d57ac572" containerName="revoke-keys"
Oct 29 22:00:19.266659 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.266420    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c512471-7b82-40c8-967f-8b96d57ac572" containerName="revoke-keys"
Oct 29 22:00:19.285601 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.285562    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.318166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.305837    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"]
Oct 29 22:00:19.608222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.584691    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.608222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.584846    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.608222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.584947    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.608222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.584991    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.608222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.585023    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.608222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.585086    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.608222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.585200    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.608222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.585281    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mphpn\" (UniqueName: \"kubernetes.io/projected/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-kube-api-access-mphpn\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.608222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.585321    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.759697 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.759657    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mphpn\" (UniqueName: \"kubernetes.io/projected/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-kube-api-access-mphpn\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.785274 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.785237    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.786017 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.785992    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.798165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.794386    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.822733 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.802300    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.856211 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.786974    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.856211 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.762518    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:19.856211 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.807565    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.856711 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.856677    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.868311 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.868184    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.868517 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.868497    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.868801 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.868761    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.874676 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.874614    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.874894 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.874876    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.880446 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.880391    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:19.959369 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.957405    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:00:19.960861 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:19.958369    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:00:20.009518 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:20.009476    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:00:20.045167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:20.043852    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mphpn\" (UniqueName: \"kubernetes.io/projected/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-kube-api-access-mphpn\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:20.045167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:20.044508    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:20.063176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:20.054742    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:20.063176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:20.054823    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:20.070791 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:20.070753    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:20.078135 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:20.078021    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:20.226918 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:20.226282    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:20.250447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:20.250386    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:20.252045 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:20.252019    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:20.518947 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:20.517300    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:20.530958 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:20.530229    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:20.549356 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:20.549255    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:20.689174 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:20.688632    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.150333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.112255    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:00:21.150333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.112717    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:00:21.150333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.129740    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:00:21.150333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.130158    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:00:21.150333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.130277    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.150333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.130444    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:00:21.150333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.130499    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:00:21.150333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.130536    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:00:21.150333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.137207    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.150333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.137510    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:00:21.279161 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.272238    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.279161 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.272626    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:00:21.318210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.300199    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:00:21.318210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.300581    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:00:21.641907 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.550433    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.641907 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.550844    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:00:21.641907 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.636693    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.662203 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.661304    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.759082 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.697201    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.759082 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.697513    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:00:21.921039 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.920948    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:21.922373 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.922338    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:21.924258 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.924224    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:22.012274 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:21.992464    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:00:22.172787 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:22.151223    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:22.297266 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:22.285003    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:00:22.394715 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:22.394666    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:22.465228 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:22.398045    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:00:22.735670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:22.721132    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:22.736738 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:22.734544    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:22.865501 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:22.865453    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:22.899188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:22.896852    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:00:22.899188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:22.896981    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:00:23.270192 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:23.226901    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:00:23.295565 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:23.234886    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:23.295565 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:23.270467    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:00:23.338173 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:00:23.335419    4550 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.342s"
Oct 29 22:00:23.338173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:23.335488    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:00:23.338173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:23.335581    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:23.338173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:23.335601    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj" event={"ID":"e8bb2593-5acc-40c9-9025-e5e6b29bd80b","Type":"ContainerStarted","Data":"79333484e6cbe877a4409469a5e6d99281375ae2beb11a3f17d6aacfdb3682f7"}
Oct 29 22:00:23.393175 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:23.392796    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:00:23.505169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:23.502867    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:23.505169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:23.504197    4550 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:23.767511 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:23.767449    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:23.769576 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:23.769541    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:00:23.875768 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:23.875723    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:00:23.914168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:23.906868    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"]
Oct 29 22:00:24.301429 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.301384    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:24.301950 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.301921    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:00:24.305463 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.305437    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:00:24.305845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.305821    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:00:24.309446 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.309418    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 29 22:00:24.309637 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.309616    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 29 22:00:24.309779 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.309761    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:00:24.408165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.399617    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:24.597356 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.597315    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:00:24.626166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.622891    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:00:24.773284 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.773241    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:24.809744 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.795175    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:24.857368 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.857269    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:00:24.857368 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.857332    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6" podUID="5a89ba55-981b-475e-a86c-833c547e65d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:24.930799 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.930324    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:00:24.969267 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:24.969211    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:25.081458 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:25.081428    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:00:25.269542 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:25.265715    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:00:25.378838 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:25.375745    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:00:25.396549 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:25.396130    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:00:25.531910 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:25.529760    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:25.815373 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:25.815268    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:25.861730 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:25.861689    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:00:25.899882 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:25.891654    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:00:25.926175 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:25.903848    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:26.187612 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:26.187570    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:00:26.238456 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:26.238393    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.337179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:26.289400    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 29 22:00:26.337179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:26.289485    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj" podUID="e8bb2593-5acc-40c9-9025-e5e6b29bd80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 29 22:00:26.376991 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:26.376950    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.438221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:26.396369    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.537569 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:26.537526    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.696571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:26.696230    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.777190 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:26.777160    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:26.778175 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:26.778130    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj" event={"ID":"e8bb2593-5acc-40c9-9025-e5e6b29bd80b","Type":"ContainerStarted","Data":"f6313501eac3e27eb555b6e5c311f8385406512bb62587cddfdbb177af894661"}
Oct 29 22:00:26.862613 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:26.862575    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:00:26.876812 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:26.872292    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.927168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:26.926680    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.962222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:26.961933    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:27.007261 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:27.007229    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:00:27.034314 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:27.032893    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj" podStartSLOduration=8.032873217 podStartE2EDuration="8.032873217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:26.443442766 +0000 UTC m=+842.600568264" watchObservedRunningTime="2026-10-29 22:00:27.032873217 +0000 UTC m=+843.189998718"
Oct 29 22:00:27.034519 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:27.034352    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:00:27.091626 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:27.091598    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:00:27.095179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:27.093430    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:00:27.115931 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:27.115902    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:00:27.123474 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:27.122499    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:00:27.255172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:27.254363    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 29 22:00:27.255172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:27.254425    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj" podUID="e8bb2593-5acc-40c9-9025-e5e6b29bd80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 29 22:00:27.998171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:27.998111    4550 scope.go:122] "RemoveContainer" containerID="a9a55f6dcac14d27f74219a5cd3bc59762fda164d068e95c442756c27d89397d"
Oct 29 22:00:28.075176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:28.072646    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:28.254476 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:28.254392    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 29 22:00:28.254634 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:28.254454    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj" podUID="e8bb2593-5acc-40c9-9025-e5e6b29bd80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 29 22:00:28.481802 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:28.481623    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:29.234383 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:29.234348    4550 generic.go:356] "Generic (PLEG): container finished" podID="5507d3ee-40cc-4da6-8ad7-64538d303eaa" containerID="81383264e57a25b4bc1cd61bcfdee0ba8088989f9137f4f529c30b7a4604cf97" exitCode=60
Oct 29 22:00:29.234771 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:29.234430    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2" event={"ID":"5507d3ee-40cc-4da6-8ad7-64538d303eaa","Type":"ContainerDied","Data":"81383264e57a25b4bc1cd61bcfdee0ba8088989f9137f4f529c30b7a4604cf97"}
Oct 29 22:00:29.234771 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:29.234500    4550 scope.go:122] "RemoveContainer" containerID="a9a55f6dcac14d27f74219a5cd3bc59762fda164d068e95c442756c27d89397d"
Oct 29 22:00:29.234898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:29.234882    4550 scope.go:122] "RemoveContainer" containerID="81383264e57a25b4bc1cd61bcfdee0ba8088989f9137f4f529c30b7a4604cf97"
Oct 29 22:00:29.235278 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:00:29.235252    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2_odh-ai-gateway-infra(5507d3ee-40cc-4da6-8ad7-64538d303eaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2" podUID="5507d3ee-40cc-4da6-8ad7-64538d303eaa"
Oct 29 22:00:29.254172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:29.254131    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 29 22:00:29.254286 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:29.254193    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj" podUID="e8bb2593-5acc-40c9-9025-e5e6b29bd80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 29 22:00:30.050567 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:30.050534    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:30.253862 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:30.253826    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 29 22:00:30.254342 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:30.253891    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj" podUID="e8bb2593-5acc-40c9-9025-e5e6b29bd80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 29 22:00:30.303571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:30.302986    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"]
Oct 29 22:00:30.513566 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:30.513542    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"
Oct 29 22:00:30.614545 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:30.614514    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5507d3ee-40cc-4da6-8ad7-64538d303eaa-kube-api-access-2fvrd\" (UniqueName: \"kubernetes.io/projected/5507d3ee-40cc-4da6-8ad7-64538d303eaa-kube-api-access-2fvrd\") pod \"5507d3ee-40cc-4da6-8ad7-64538d303eaa\" (UID: \"5507d3ee-40cc-4da6-8ad7-64538d303eaa\") "
Oct 29 22:00:30.614699 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:30.614679    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5507d3ee-40cc-4da6-8ad7-64538d303eaa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5507d3ee-40cc-4da6-8ad7-64538d303eaa-maas-api-service-ca\") pod \"5507d3ee-40cc-4da6-8ad7-64538d303eaa\" (UID: \"5507d3ee-40cc-4da6-8ad7-64538d303eaa\") "
Oct 29 22:00:30.615035 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:30.615003    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5507d3ee-40cc-4da6-8ad7-64538d303eaa-maas-api-service-ca" pod "5507d3ee-40cc-4da6-8ad7-64538d303eaa" (UID: "5507d3ee-40cc-4da6-8ad7-64538d303eaa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:00:30.615271 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:30.615253    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5507d3ee-40cc-4da6-8ad7-64538d303eaa-maas-api-service-ca\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:00:30.616923 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:30.616786    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5507d3ee-40cc-4da6-8ad7-64538d303eaa-kube-api-access-2fvrd" pod "5507d3ee-40cc-4da6-8ad7-64538d303eaa" (UID: "5507d3ee-40cc-4da6-8ad7-64538d303eaa"). InnerVolumeSpecName "kube-api-access-2fvrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:00:30.716715 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:30.716676    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fvrd\" (UniqueName: \"kubernetes.io/projected/5507d3ee-40cc-4da6-8ad7-64538d303eaa-kube-api-access-2fvrd\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:00:31.251740 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:31.251714    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"
Oct 29 22:00:31.251934 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:31.251744    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2" event={"ID":"5507d3ee-40cc-4da6-8ad7-64538d303eaa","Type":"ContainerDied","Data":"8f227873ac1f83dd6ac82f5fef1dd736ddddd1acb316cf286b040919c3831385"}
Oct 29 22:00:31.251934 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:31.251810    4550 scope.go:122] "RemoveContainer" containerID="81383264e57a25b4bc1cd61bcfdee0ba8088989f9137f4f529c30b7a4604cf97"
Oct 29 22:00:31.253610 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:31.253586    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 29 22:00:31.253815 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:31.253789    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj" podUID="e8bb2593-5acc-40c9-9025-e5e6b29bd80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 29 22:00:31.334391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:31.334267    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"]
Oct 29 22:00:31.337711 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:31.337687    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888520-vp8g2"]
Oct 29 22:00:31.875479 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:31.874477    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:32.004477 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:32.004443    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5507d3ee-40cc-4da6-8ad7-64538d303eaa" path="/var/lib/kubelet/pods/5507d3ee-40cc-4da6-8ad7-64538d303eaa/volumes"
Oct 29 22:00:32.011247 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:32.011224    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:32.023534 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:32.023510    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:00:32.254345 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:32.254262    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 29 22:00:32.254345 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:32.254317    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj" podUID="e8bb2593-5acc-40c9-9025-e5e6b29bd80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 29 22:00:33.513417 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:33.513375    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:33.684167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:33.682638    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:33.798684 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:33.798548    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:33.800063 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:33.800037    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:33.806165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:33.804690    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:00:35.519796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:35.519762    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:37.216354 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:37.216270    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:38.823794 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:38.823756    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:39.296075 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.295991    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"]
Oct 29 22:00:39.297370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.297342    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5507d3ee-40cc-4da6-8ad7-64538d303eaa" containerName="cleanup"
Oct 29 22:00:39.297541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.297525    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5507d3ee-40cc-4da6-8ad7-64538d303eaa" containerName="cleanup"
Oct 29 22:00:39.297654 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.297632    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5507d3ee-40cc-4da6-8ad7-64538d303eaa" containerName="cleanup"
Oct 29 22:00:39.297654 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.297650    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5507d3ee-40cc-4da6-8ad7-64538d303eaa" containerName="cleanup"
Oct 29 22:00:39.297841 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.297695    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5507d3ee-40cc-4da6-8ad7-64538d303eaa" containerName="cleanup"
Oct 29 22:00:39.297841 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.297704    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5507d3ee-40cc-4da6-8ad7-64538d303eaa" containerName="cleanup"
Oct 29 22:00:39.297948 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.297892    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5507d3ee-40cc-4da6-8ad7-64538d303eaa" containerName="cleanup"
Oct 29 22:00:39.297948 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.297913    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5507d3ee-40cc-4da6-8ad7-64538d303eaa" containerName="cleanup"
Oct 29 22:00:39.303103 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.303077    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.335431 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.335401    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a3c02e\"/\"distinct-w1-a3c02e-kserve-self-signed-certs\""
Oct 29 22:00:39.337079 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.337047    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"]
Oct 29 22:00:39.368419 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.368355    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:39.447540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.447508    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-dshm\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.447788 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.447642    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-tmp-dir\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.447788 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.447698    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nzfw\" (UniqueName: \"kubernetes.io/projected/62e5ef97-3c39-467a-8bf8-60d549360d47-kube-api-access-4nzfw\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.447788 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.447749    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-home\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.448050 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.447896    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-model-cache\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.448050 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.447918    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e5ef97-3c39-467a-8bf8-60d549360d47-tls-certs\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.564212 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.555266    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nzfw\" (UniqueName: \"kubernetes.io/projected/62e5ef97-3c39-467a-8bf8-60d549360d47-kube-api-access-4nzfw\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.564212 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.555689    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-home\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.564212 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.555893    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-model-cache\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.564212 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.555927    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e5ef97-3c39-467a-8bf8-60d549360d47-tls-certs\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.564212 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.555953    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-dshm\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.564212 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.556068    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-tmp-dir\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.564212 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.556547    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-tmp-dir\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.564212 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.557223    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-home\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.564212 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.557650    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-model-cache\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.590216 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.581048    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e5ef97-3c39-467a-8bf8-60d549360d47-tls-certs\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.599406 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.599023    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nzfw\" (UniqueName: \"kubernetes.io/projected/62e5ef97-3c39-467a-8bf8-60d549360d47-kube-api-access-4nzfw\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.621186 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.618541    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-dshm\") pod \"distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.640739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.640227    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:39.991614 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:39.991589    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"]
Oct 29 22:00:39.993195 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:00:39.993168    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62e5ef97_3c39_467a_8bf8_60d549360d47.slice/crio-af7a5b87d7b0fce76571bd2b149c4d7fd97ac711523e1b0275b996c332252c7d WatchSource:0}: Error finding container af7a5b87d7b0fce76571bd2b149c4d7fd97ac711523e1b0275b996c332252c7d: Status 404 returned error can't find the container with id af7a5b87d7b0fce76571bd2b149c4d7fd97ac711523e1b0275b996c332252c7d
Oct 29 22:00:40.208605 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.208568    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"]
Oct 29 22:00:40.209566 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.209538    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5507d3ee-40cc-4da6-8ad7-64538d303eaa" containerName="cleanup"
Oct 29 22:00:40.213396 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.213373    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.248160 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.248048    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"]
Oct 29 22:00:40.340860 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.340823    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" event={"ID":"62e5ef97-3c39-467a-8bf8-60d549360d47","Type":"ContainerStarted","Data":"98453a51df008f8069c330ac05ec7a07e6a634874aef6d40f4084d4a261ba3fc"}
Oct 29 22:00:40.341028 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.340877    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" event={"ID":"62e5ef97-3c39-467a-8bf8-60d549360d47","Type":"ContainerStarted","Data":"af7a5b87d7b0fce76571bd2b149c4d7fd97ac711523e1b0275b996c332252c7d"}
Oct 29 22:00:40.384290 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.384258    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.384471 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.384314    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.384533 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.384462    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.384533 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.384502    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.384631 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.384544    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tchb9\" (UniqueName: \"kubernetes.io/projected/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-kube-api-access-tchb9\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.384666 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.384632    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.384705 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.384669    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.384780 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.384757    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.384911 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.384893    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.394411 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.394364    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podStartSLOduration=1.39435253 podStartE2EDuration="1.39435253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:40.393486804 +0000 UTC m=+856.550612303" watchObservedRunningTime="2026-10-29 22:00:40.39435253 +0000 UTC m=+856.551478029"
Oct 29 22:00:40.485754 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.485719    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.485754 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.485753    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.485999 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.485941    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tchb9\" (UniqueName: \"kubernetes.io/projected/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-kube-api-access-tchb9\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.486065 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.486015    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.486065 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.486057    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.486205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.486061    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.486205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.486119    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.486205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.486132    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.486381 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.486279    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.486484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.486384    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.486484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.486458    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.486580 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.486534    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.486793 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.486772    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.487074 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.487053    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.488287 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.488267    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.488767 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.488750    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.512486 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.511716    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tchb9\" (UniqueName: \"kubernetes.io/projected/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-kube-api-access-tchb9\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.512486 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.511942    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.531533 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.531496    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:40.642042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.641455    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:40.657750 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.657696    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:40.860977 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:00:40.860947    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c9a8e5_9324_4ded_bdb1_03e9e9ff7da7.slice/crio-72d2811493300928f5b9dec97841967ef972876cf1889117295feb9f67916921 WatchSource:0}: Error finding container 72d2811493300928f5b9dec97841967ef972876cf1889117295feb9f67916921: Status 404 returned error can't find the container with id 72d2811493300928f5b9dec97841967ef972876cf1889117295feb9f67916921
Oct 29 22:00:40.861107 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.860994    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"]
Oct 29 22:00:40.863846 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.863819    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:40.863939 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.863882    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:40.863939 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:40.863914    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:41.348317 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:41.348282    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7" event={"ID":"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7","Type":"ContainerStarted","Data":"ff51d0482e228e94bcf6b6582abd59d7feddd9cbede9ab8a260a17c735ebeef9"}
Oct 29 22:00:41.348703 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:41.348339    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7" event={"ID":"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7","Type":"ContainerStarted","Data":"72d2811493300928f5b9dec97841967ef972876cf1889117295feb9f67916921"}
Oct 29 22:00:41.427453 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:41.427398    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7" podStartSLOduration=1.42738423 podStartE2EDuration="1.42738423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:41.413212087 +0000 UTC m=+857.570337586" watchObservedRunningTime="2026-10-29 22:00:41.42738423 +0000 UTC m=+857.584509728"
Oct 29 22:00:41.491186 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:00:41.491134    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:00:41.532664 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:41.532636    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:41.534499 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:41.534470    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:00:41.534666 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:41.534525    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7" podUID="a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:00:42.445845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:42.445813    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:42.532485 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:42.532450    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:00:42.532695 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:42.532505    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7" podUID="a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:00:43.532862 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:43.532820    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:00:43.533284 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:43.532890    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7" podUID="a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:00:44.183166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:44.161533    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:44.532081 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:44.531997    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:00:44.532081 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:44.532054    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7" podUID="a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:00:45.532151 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:45.532111    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:00:45.532554 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:45.532213    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7" podUID="a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:00:46.166598 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:46.166561    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:46.532968 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:46.532882    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:00:46.532968 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:46.532938    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7" podUID="a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:00:47.532754 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:47.532714    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:00:47.533071 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:47.533032    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7" podUID="a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:00:47.941990 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:47.941960    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:48.537297 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:48.537269    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:48.537706 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:48.537550    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:48.538517 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:48.538495    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:00:49.577319 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:49.577265    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:49.641296 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:49.641265    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:49.655867 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:49.655835    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:50.450523 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.450494    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:00:50.596910 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.596874    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"]
Oct 29 22:00:50.609339 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.609311    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.630045 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.630017    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"]
Oct 29 22:00:50.759405 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.759330    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.759549 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.759456    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.759590 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.759546    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sp9s\" (UniqueName: \"kubernetes.io/projected/d6233b12-31ad-41d0-bb31-12b2f71161bb-kube-api-access-6sp9s\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.759590 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.759574    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6233b12-31ad-41d0-bb31-12b2f71161bb-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.759675 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.759605    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.759717 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.759696    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.759767 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.759726    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.759812 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.759779    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.759812 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.759802    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.861389 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.861352    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.861576 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.861447    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sp9s\" (UniqueName: \"kubernetes.io/projected/d6233b12-31ad-41d0-bb31-12b2f71161bb-kube-api-access-6sp9s\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.861576 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.861470    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6233b12-31ad-41d0-bb31-12b2f71161bb-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.861576 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.861504    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.861576 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.861543    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.861807 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.861579    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.861807 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.861609    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.861807 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.861642    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.861953 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.861811    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.861953 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.861840    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.861953 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.861903    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.862112 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.862089    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.862188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.862110    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.862263 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.862245    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6233b12-31ad-41d0-bb31-12b2f71161bb-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.864049 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.864025    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.864202 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.864184    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.885679 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.885651    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sp9s\" (UniqueName: \"kubernetes.io/projected/d6233b12-31ad-41d0-bb31-12b2f71161bb-kube-api-access-6sp9s\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.885789 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.885740    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:50.927777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:50.927756    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:51.116701 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:51.116669    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:51.257260 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:00:51.257224    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6233b12_31ad_41d0_bb31_12b2f71161bb.slice/crio-151a2a779943e53b0b229fcd3574ba07f9014259a2144091b0a5de30d553a3da WatchSource:0}: Error finding container 151a2a779943e53b0b229fcd3574ba07f9014259a2144091b0a5de30d553a3da: Status 404 returned error can't find the container with id 151a2a779943e53b0b229fcd3574ba07f9014259a2144091b0a5de30d553a3da
Oct 29 22:00:51.259456 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:51.259421    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:51.259583 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:51.259489    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:51.259583 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:51.259519    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:00:51.261503 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:51.261476    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"]
Oct 29 22:00:51.448484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:51.448386    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp" event={"ID":"d6233b12-31ad-41d0-bb31-12b2f71161bb","Type":"ContainerStarted","Data":"18c2bba4296d509437428577b04f13ff465481fe334118306f474a9e8c5183b9"}
Oct 29 22:00:51.448484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:51.448449    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp" event={"ID":"d6233b12-31ad-41d0-bb31-12b2f71161bb","Type":"ContainerStarted","Data":"151a2a779943e53b0b229fcd3574ba07f9014259a2144091b0a5de30d553a3da"}
Oct 29 22:00:51.498120 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:51.498050    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp" podStartSLOduration=1.49803787 podStartE2EDuration="1.49803787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:51.495362686 +0000 UTC m=+867.652488207" watchObservedRunningTime="2026-10-29 22:00:51.49803787 +0000 UTC m=+867.655163368"
Oct 29 22:00:51.928496 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:51.928460    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:52.835060 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:52.835015    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:52.939056 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:52.939019    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:00:52.939600 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:52.939136    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp" podUID="d6233b12-31ad-41d0-bb31-12b2f71161bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 29 22:00:53.928657 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:53.928621    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:00:53.928834 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:53.928694    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp" podUID="d6233b12-31ad-41d0-bb31-12b2f71161bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 29 22:00:54.650773 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:54.650735    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:54.928886 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:54.928812    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:00:54.928886 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:54.928864    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp" podUID="d6233b12-31ad-41d0-bb31-12b2f71161bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 29 22:00:55.928889 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:55.928857    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:00:55.929307 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:55.928908    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp" podUID="d6233b12-31ad-41d0-bb31-12b2f71161bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 29 22:00:56.442894 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:56.442856    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:56.928527 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:56.928489    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:00:56.928702 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:56.928552    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp" podUID="d6233b12-31ad-41d0-bb31-12b2f71161bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 29 22:00:57.928724 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:57.928690    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:00:57.929125 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:57.928749    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp" podUID="d6233b12-31ad-41d0-bb31-12b2f71161bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 29 22:00:58.335756 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:58.335716    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:00:58.933252 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:58.933219    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:58.933649 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:58.933625    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:00:58.934364 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:00:58.934346    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:01:00.304521 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:00.304486    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:01.958570 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:01.958528    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:03.535781 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:03.535740    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:05.269983 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:05.269952    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:07.059950 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:07.059910    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:08.973628 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:08.973592    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:09.106057 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:09.106020    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:09.236196 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:09.236056    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:10.264056 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:10.264017    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:10.383676 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:10.383642    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:10.513165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:10.507589    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:10.891485 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:10.891451    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:11.449129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:11.449096    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:11.491387 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:01:11.491355    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:01:12.525020 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:12.524973    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:14.224910 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:14.224876    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:15.921418 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:15.921382    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:17.648036 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:17.648001    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:19.312675 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:19.312631    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:21.034155 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:21.034108    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:22.398283 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:22.398244    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:22.739337 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:22.739252    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:24.250937 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:24.250905    4550 scope.go:122] "RemoveContainer" containerID="772a0e45c2d25e291f06f0c470f75856acd867499772c140bc79fff31279ba9e"
Oct 29 22:01:24.756538 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:24.756500    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:27.247166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:27.246838    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:27.306381 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:27.306313    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:27.307473 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:27.307441    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:27.394185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:27.393224    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:27.401248 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:27.401210    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:27.443913 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:27.443871    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:27.826176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:27.822256    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:27.843560 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:27.843503    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:27.882288 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:27.879214    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:28.015411 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.015363    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:28.244435 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.244312    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:28.353218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.322246    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:28.408731 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.408680    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:01:28.409372 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.409350    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:01:28.409597 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.409582    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:01:28.482269 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.411626    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:28.512468 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.440133    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:01:28.537170 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.529882    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:01:28.567829 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.567794    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:01:28.632237 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.444368    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:28.826229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.721224    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded"
Oct 29 22:01:28.826229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.780742    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:01:28.826229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.821248    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:01:28.826229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.822534    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:28.827634 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.826752    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:01:28.901392 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.898227    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:01:28.901392 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:28.900350    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.033158 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.033093    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.034582 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.034557    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:01:29.297221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.296841    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.301166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.298272    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:01:29.431771 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.431727    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:01:29.455166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.432653    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:01:29.500476 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.500430    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:01:29.538521 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.538468    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:01:29.717001 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.674294    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:29.717001 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.674371    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.795277 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.795230    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:01:29.825629 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.825251    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:01:29.860166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.833714    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:01:29.860166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.834126    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:01:29.909290 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.909240    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.981226 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.958461    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:01:30.039170 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.911212    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:30.063595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:29.909529    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:01:30.069010 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:30.038369    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:01:30.307231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:30.301278    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:30.499157 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:30.440780    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:30.588527 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:30.588467    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:30.600222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:30.590400    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:30.600222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:30.590967    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:30.600222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:30.594711    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:01:30.600222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:30.595347    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 29 22:01:31.233975 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:30.966297    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:01:31.233975 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:30.966704    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:01:31.233975 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:31.030932    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:31.256201 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:31.234537    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:01:31.256201 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:31.234990    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:01:31.310379 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:31.309696    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.504681 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:31.504566    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.625685 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:31.515195    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded"
Oct 29 22:01:31.648191 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:31.635243    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.657170 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:31.653364    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:01:31.888226 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:31.597502    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.921203 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:31.890110    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.921597 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:31.921522    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:01:31.954066 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:31.953987    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:01:31.977134 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:31.977085    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.978888 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:31.978834    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:01:32.037417 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:01:32.037379    4550 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.056s"
Oct 29 22:01:32.062682 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.062655    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:01:32.102433 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.102405    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:01:32.241792 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.241694    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.263389 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.252310    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:01:32.351093 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.335319    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.451166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.442581    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:01:32.478807 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.471299    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:01:32.505650 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.505543    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.536886 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.535508    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:32.706591 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.706541    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.880046 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.880002    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.899355 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.899315    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:01:32.913419 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.902473    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.950796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.925432    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.950796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.926748    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:32.950796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.950222    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.966245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.954458    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.988699 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.988647    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.989079 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:32.988952    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.009778 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.009733    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded"
Oct 29 22:01:33.234230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.230815    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.253177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.252812    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.253372 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.253181    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:01:33.355169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.350270    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.475167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.475059    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:01:33.491842 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.475479    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:01:33.542171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.515091    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:01:33.681163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.605269    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:33.681163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.654358    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.781349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.781238    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.856787 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.856740    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.919040 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.918997    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.931465 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.931424    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.932023 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.931682    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.969615 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:33.969245    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.027192 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.004669    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.081809 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.060368    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:01:34.131554 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.129098    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:01:34.389218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.378359    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.389218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.378670    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:01:34.389218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.379272    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.417771 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.417739    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:01:34.659783 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.659697    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.720159 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.705953    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:01:34.783531 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.781329    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:01:34.783531 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.783030    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.926241 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.923097    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.932610 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.932534    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:01:34.933168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.933122    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:01:34.971226 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.970774    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:01:34.971602 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:34.971567    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.018180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:35.011365    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.090171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:35.080473    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:01:35.101558 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:35.101516    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.309236 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:35.292102    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:01:35.487169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:35.445589    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:01:35.521175 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:35.489734    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:35.549099 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:35.516527    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:01:35.701859 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:35.701828    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:01:35.789320 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:35.789290    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:01:35.949806 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:35.949756    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.988743 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:35.950079    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:36.016287 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:36.014224    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:36.055184 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:36.054385    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:01:36.057706 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:36.057688    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:01:36.061342 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:36.061321    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:01:37.352881 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:37.352839    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:39.012010 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:39.011978    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:40.743168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:40.742384    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:41.492497 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:01:41.492463    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:01:42.529605 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:42.529572    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:44.212823 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:44.212794    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:45.981555 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:45.981513    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:46.655697 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:46.655662    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:47.729543 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:47.729512    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:49.291684 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:49.291645    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:51.000359 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:51.000326    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:52.524309 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:52.524253    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:54.162173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:54.162075    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:55.415338 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:55.415306    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:55.735428 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:55.735339    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:55.895725 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:55.895691    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:56.117169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:56.104075    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:57.396789 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:57.396754    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:01:59.745489 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:01:59.745456    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:01.013617 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:01.013588    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:01.272319 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:01.272214    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:01.392096 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:01.392056    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:02.831624 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:02.831593    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:04.564995 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:04.564956    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:06.277957 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:06.277920    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:07.918446 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:07.918410    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:09.469218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:09.469185    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:11.139412 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:11.139382    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:11.492196 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:02:11.492094    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:02:11.576093 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:11.576064    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:11.701594 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:11.701563    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:13.075747 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:13.075711    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:13.207641 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:13.207587    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:13.330851 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:13.330766    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:15.362514 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:15.362475    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:17.124218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:17.124184    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:18.725816 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:18.725783    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:20.415741 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:20.415703    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:22.130394 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:22.130355    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:22.359375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:22.359333    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:22.913169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:22.913087    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:23.287804 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:23.287721    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:24.068724 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:24.068670    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:25.738342 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:25.738310    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:26.942959 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:26.942928    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:27.063276 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:27.063237    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:27.660616 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:27.660552    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:28.595580 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:28.595509    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:28.738512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:28.738475    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:29.533941 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:29.533908    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:31.383816 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:31.383782    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:33.134202 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:33.134168    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:34.830034 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:34.830005    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:34.977846 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:34.977813    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:36.462233 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:36.462194    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:36.983740 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:36.983705    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"]
Oct 29 22:02:36.987893 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:36.987873    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.009698 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.009667    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"]
Oct 29 22:02:37.017074 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.017045    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-751a7416\"/\"default-dockercfg-qvmvs\""
Oct 29 22:02:37.017074 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.017060    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-751a7416\"/\"e2e-lifecycle-model-751a7416-kserve-self-signed-certs\""
Oct 29 22:02:37.019313 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.019288    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-751a7416\"/\"kube-root-ca.crt\""
Oct 29 22:02:37.019313 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.019298    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-751a7416\"/\"openshift-service-ca.crt\""
Oct 29 22:02:37.046186 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.046135    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:37.179210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.177571    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-tmp-dir\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.187273 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.187186    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-dshm\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.187472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.187295    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f0b9a2-8e74-4f05-9754-5f3b18440530-tls-certs\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.187472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.187386    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-home\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.187472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.187443    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-model-cache\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.187682 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.187627    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbb6c\" (UniqueName: \"kubernetes.io/projected/89f0b9a2-8e74-4f05-9754-5f3b18440530-kube-api-access-fbb6c\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.295855 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.295756    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-tmp-dir\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.296808 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.296777    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-dshm\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.297449 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.297428    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f0b9a2-8e74-4f05-9754-5f3b18440530-tls-certs\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.297618 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.297603    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-home\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.297756 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.297741    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-model-cache\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.297994 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.297978    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbb6c\" (UniqueName: \"kubernetes.io/projected/89f0b9a2-8e74-4f05-9754-5f3b18440530-kube-api-access-fbb6c\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.298714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.296618    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-tmp-dir\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.299431 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.299408    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-home\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.299816 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.299794    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-model-cache\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.312907 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.312870    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f0b9a2-8e74-4f05-9754-5f3b18440530-tls-certs\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.325211 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.325167    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-dshm\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.326284 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.326223    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbb6c\" (UniqueName: \"kubernetes.io/projected/89f0b9a2-8e74-4f05-9754-5f3b18440530-kube-api-access-fbb6c\") pod \"e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") " pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.603123 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.603094    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:37.911921 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:02:37.911889    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89f0b9a2_8e74_4f05_9754_5f3b18440530.slice/crio-2fa28136c8de0829e805d6ff7267e66bfdc8295c46b1d05e93988ac45278d326 WatchSource:0}: Error finding container 2fa28136c8de0829e805d6ff7267e66bfdc8295c46b1d05e93988ac45278d326: Status 404 returned error can't find the container with id 2fa28136c8de0829e805d6ff7267e66bfdc8295c46b1d05e93988ac45278d326
Oct 29 22:02:37.912374 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:37.912353    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"]
Oct 29 22:02:38.061302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:38.061261    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" event={"ID":"89f0b9a2-8e74-4f05-9754-5f3b18440530","Type":"ContainerStarted","Data":"5723829c38825fe65209d59004490fcef8a0e4e1d350287467f56647da3818b0"}
Oct 29 22:02:38.061476 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:38.061310    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" event={"ID":"89f0b9a2-8e74-4f05-9754-5f3b18440530","Type":"ContainerStarted","Data":"2fa28136c8de0829e805d6ff7267e66bfdc8295c46b1d05e93988ac45278d326"}
Oct 29 22:02:38.105956 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:38.105908    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podStartSLOduration=2.105894021 podStartE2EDuration="2.105894021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:38.103701931 +0000 UTC m=+974.260827442" watchObservedRunningTime="2026-10-29 22:02:38.105894021 +0000 UTC m=+974.263019519"
Oct 29 22:02:38.344775 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:38.344736    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:38.603656 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:38.603580    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:39.979379 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:39.979347    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:41.500208 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:02:41.500167    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:02:41.590857 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.590824    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"]
Oct 29 22:02:41.594688 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.594668    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:41.618668 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.618605    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a9f9aa-serving-cert\""
Oct 29 22:02:41.623730 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.623698    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"]
Oct 29 22:02:41.689344 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.689312    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-metrics-certs\") pod \"maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:41.689504 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.689379    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-openshift-service-ca\") pod \"maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:41.689606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.689582    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-maas-api-tls\") pod \"maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:41.689776 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.689757    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvqcn\" (UniqueName: \"kubernetes.io/projected/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-kube-api-access-zvqcn\") pod \"maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:41.735733 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.735701    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:41.795733 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.792062    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvqcn\" (UniqueName: \"kubernetes.io/projected/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-kube-api-access-zvqcn\") pod \"maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:41.795733 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.792285    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-metrics-certs\") pod \"maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:41.795733 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.792354    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-openshift-service-ca\") pod \"maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:41.795733 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.792459    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-maas-api-tls\") pod \"maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:41.797256 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.797227    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-openshift-service-ca\") pod \"maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:41.801947 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.801917    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-metrics-certs\") pod \"maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:41.803794 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.803768    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-maas-api-tls\") pod \"maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:41.828926 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.828890    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvqcn\" (UniqueName: \"kubernetes.io/projected/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-kube-api-access-zvqcn\") pod \"maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:41.864076 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.864048    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:41.910697 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:41.910665    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:42.230200 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:42.230169    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"]
Oct 29 22:02:42.230557 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:02:42.230532    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ec2d58_3402_4880_86e8_ab9fdbc4a9c8.slice/crio-99202135ddaf32eca3a91695114deddae670142eb2299edb180bf05dfe0831dc WatchSource:0}: Error finding container 99202135ddaf32eca3a91695114deddae670142eb2299edb180bf05dfe0831dc: Status 404 returned error can't find the container with id 99202135ddaf32eca3a91695114deddae670142eb2299edb180bf05dfe0831dc
Oct 29 22:02:43.096103 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:43.096061    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb" event={"ID":"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8","Type":"ContainerStarted","Data":"ea55eec8903227761b24567c708a0bb8540d6335c26c277812c77251796c04e9"}
Oct 29 22:02:43.096485 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:43.096120    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb" event={"ID":"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8","Type":"ContainerStarted","Data":"99202135ddaf32eca3a91695114deddae670142eb2299edb180bf05dfe0831dc"}
Oct 29 22:02:43.096485 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:43.096162    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:43.145473 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:43.145404    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb" podStartSLOduration=1.693921904 podStartE2EDuration="2.145389015s" totalImagesPullingTime="451.467111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:43.141515105 +0000 UTC m=+979.298640604" watchObservedRunningTime="2026-10-29 22:02:43.145389015 +0000 UTC m=+979.302514519"
Oct 29 22:02:43.590657 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:43.590623    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:45.092601 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:45.092572    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:46.714474 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:46.714443    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:47.604203 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:47.604172    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:47.616967 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:47.616939    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:48.138413 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:48.138383    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:48.381986 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:48.381953    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:49.126163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:49.125481    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:02:50.038498 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:50.038470    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:52.285352 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:52.285290    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:52.959883 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:52.958310    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:02:52.973072 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:52.972275    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.024801 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.024755    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.157384 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.154207    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.217353 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.213336    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.234163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.223398    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.286166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.270923    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.367279 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.367235    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.449224 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.449180    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.509222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.492229    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.560785 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.559249    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.736763 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.736719    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:02:53.819806 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.819711    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.828300 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.828217    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.903278 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.903229    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.932287 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.932244    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:02:53.967498 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.965450    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:02:53.967498 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.965777    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:02:53.983810 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.982534    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.983810 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:53.982853    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:02:54.203803 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.193633    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.224167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.221263    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.290306 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.290253    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.388405 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.369251    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.391878 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.391846    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:02:54.395165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.392065    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"]
Oct 29 22:02:54.425168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.414622    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"]
Oct 29 22:02:54.425168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.415046    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:54.425168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.415623    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:02:54.425168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.415703    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:02:54.425168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.416364    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:02:54.472236 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.434586    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jjwwj\""
Oct 29 22:02:54.472236 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.435474    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 29 22:02:54.472236 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.435531    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 29 22:02:54.472236 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.435599    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 29 22:02:54.472236 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.450168    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.472236 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.450575    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:02:54.505694 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.493280    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.505694 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.493685    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:02:54.794375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.579100    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34be22aa-35ff-4fb0-90d8-f2a523511916-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:54.794375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.579199    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/34be22aa-35ff-4fb0-90d8-f2a523511916-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:54.794375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.579502    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkmdj\" (UniqueName: \"kubernetes.io/projected/34be22aa-35ff-4fb0-90d8-f2a523511916-kube-api-access-tkmdj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:54.794375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.579673    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/34be22aa-35ff-4fb0-90d8-f2a523511916-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:54.794375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.579712    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/34be22aa-35ff-4fb0-90d8-f2a523511916-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:54.794375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.636270    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:54.794375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.636337    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk" podUID="2913dd75-7d51-4ee4-b705-f6a7396d75de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.794375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.637994    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.794375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.686276    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:02:54.794375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.743477    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.983215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.917009    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:02:55.333168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.932742    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:02:55.333168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.933939    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.936235    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:55.440397    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:55.440607    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:55.440678    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:55.442039    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:55.442389    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:55.442693    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:55.442960    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.985237    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.990657    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:55.605053    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6" podUID="5a89ba55-981b-475e-a86c-833c547e65d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:54.992547    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:55.195270    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/34be22aa-35ff-4fb0-90d8-f2a523511916-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:55.605600    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/34be22aa-35ff-4fb0-90d8-f2a523511916-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:55.606963    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:55.607543    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:02:55.679714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:55.647696    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:55.781306 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:55.781256    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.309850 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.297251    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.338180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.329300    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.338180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.329480    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:02:56.464172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.462692    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:02:56.464496 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.464452    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:02:56.648167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.464661    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:02:56.674213 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.464688    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:02:56.674926 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.464725    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:02:56.732169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.517258    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" podUID="93b4187a-1551-4cf8-b807-d057baf03920" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.732560 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.732508    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.733029 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.548076    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34be22aa-35ff-4fb0-90d8-f2a523511916-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:56.733258 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.733236    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/34be22aa-35ff-4fb0-90d8-f2a523511916-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:56.733696 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.733674    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkmdj\" (UniqueName: \"kubernetes.io/projected/34be22aa-35ff-4fb0-90d8-f2a523511916-kube-api-access-tkmdj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:56.972463 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.972370    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/34be22aa-35ff-4fb0-90d8-f2a523511916-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:56.972723 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.972704    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/34be22aa-35ff-4fb0-90d8-f2a523511916-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:56.972859 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.972845    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/34be22aa-35ff-4fb0-90d8-f2a523511916-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:57.003123 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.003087    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/34be22aa-35ff-4fb0-90d8-f2a523511916-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:57.004495 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.850346    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:02:57.042171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.009258    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/34be22aa-35ff-4fb0-90d8-f2a523511916-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:57.237232 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.855200    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Oct 29 22:02:57.237232 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.855236    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:57.237232 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.234373    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h" podUID="be3e55fd-83b9-4c52-a2a9-28da7d04477e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.237232 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.598388    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34be22aa-35ff-4fb0-90d8-f2a523511916-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:57.237232 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.855167    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.237232 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.855092    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.237232 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.761765    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:02:57.237232 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:56.797354    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.358089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.353682    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.358089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.354019    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:02:57.358089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.354292    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:02:57.379169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.362347    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/34be22aa-35ff-4fb0-90d8-f2a523511916-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:57.379493 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.367805    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/34be22aa-35ff-4fb0-90d8-f2a523511916-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:57.399166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.380451    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkmdj\" (UniqueName: \"kubernetes.io/projected/34be22aa-35ff-4fb0-90d8-f2a523511916-kube-api-access-tkmdj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:57.440476 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.440401    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/34be22aa-35ff-4fb0-90d8-f2a523511916-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:57.443281 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.443252    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/34be22aa-35ff-4fb0-90d8-f2a523511916-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zlz8n\" (UID: \"34be22aa-35ff-4fb0-90d8-f2a523511916\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:57.484169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.483259    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:57.484169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.483335    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.509222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.484952    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.657169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.626205    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.658077 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.657438    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.658077 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.657815    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.709177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.708266    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:02:57.736166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.735902    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.747167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.737436    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:02:57.886208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:57.865665    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.017627 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.017571    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.146820 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.121719    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.263819 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.263764    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.264486 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.264059    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:02:58.265726 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.264089    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:02:58.368776 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.368722    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.370197 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.369364    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.370197 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.369398    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.523363 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.523250    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.622462 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.622418    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": context deadline exceeded"
Oct 29 22:02:58.630278 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.630238    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.658299 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.658252    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.659441 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.659396    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.710296 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.710241    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.712220 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.712186    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.739696 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:58.739654    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.034509 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.034463    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.286170 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.276240    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.286170 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.276661    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.318248 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.317867    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.385873 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.385464    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:02:59.402166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.386790    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.402166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.387013    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.520234 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.519589    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:59.520234 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.519654    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf" podUID="db332eb5-5ced-43e9-953b-5d94aa37d706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.646185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.646096    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.688834 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.646456    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.689575 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.661222    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.720116 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.661261    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:02:59.720116 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.718573    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.766470 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.740231    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:02:59.826468 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:02:59.823481    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"]
Oct 29 22:03:00.064826 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.036665    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:03:00.086631 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.086250    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:00.086631 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.086313    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp" podUID="d6233b12-31ad-41d0-bb31-12b2f71161bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.143394 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.143358    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:00.143905 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.143848    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49" podUID="37c48198-27e2-4aa6-86e7-389bd62a2666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.150629 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.150492    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n" event={"ID":"34be22aa-35ff-4fb0-90d8-f2a523511916","Type":"ContainerStarted","Data":"fd03e118fdafc24ba015499599c20212c5314a961cb2c67d2c9b535b15130c37"}
Oct 29 22:03:00.280798 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.278257    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.280798 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.278445    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:03:00.319902 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.319208    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.398229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.397628    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.398229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.398048    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:03:00.398791 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.398409    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:03:00.498168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.493356    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.531294 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.531245    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.659374 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.659272    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.659912 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.659599    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:03:00.690500 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.690448    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.691770 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.691737    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.692343 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.692123    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.721283 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.721245    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:03:00.742279 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:00.742223    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.038353 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:01.038253    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.284062 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:01.281247    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.284062 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:01.281653    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.343873 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:01.342981    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:03:01.351477 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:01.349319    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:03:01.352161 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:01.351784    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:03:01.397569 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:01.397539    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:03:01.400076 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:01.400044    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.592165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:01.571956    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.834655 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:01.821548    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.835166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:01.822777    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.835865 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:01.824577    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.840377 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:01.824604    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:03:01.840377 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:01.837229    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.899050 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:01.891256    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.039320 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:02.039242    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.057172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:02.056247    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:02.057172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:02.056309    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.222609 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:02.222241    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:02.222609 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:02.222302    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.283309 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:02.282237    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.348859 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:02.347155    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.399374 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:02.394707    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:03:02.411292 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:02.411210    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.789740 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:02.789704    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:03:02.859275 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:02.859228    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:03:02.860913 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:02.860086    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.860913 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:02.860426    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.860913 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:02.860670    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:03:02.924176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:02.906708    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.005170 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.003546    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.041027 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.040241    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:03:03.082707 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.082679    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:03:03.092219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.091376    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:03:03.166402 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.166374    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:03:03.260281 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.260235    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 29 22:03:03.332073 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.332017    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.387201 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.387099    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:03:03.411427 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.410216    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:03:03.555128 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.555024    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:03.569343 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.569311    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:03.569600 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.569562    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.686345 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.686262    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:03:03.869088 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.865761    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.869088 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.866106    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.869088 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.865522    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:03:03.936303 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:03.912993    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:03:04.004381 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:04.004311    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:03:05.439550 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:05.439517    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:05.931413 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:05.931382    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:03:05.931516 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:05.931444    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:03:06.360756 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:06.360719    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n" event={"ID":"34be22aa-35ff-4fb0-90d8-f2a523511916","Type":"ContainerStarted","Data":"7954e40752cc9df065bb955c0dc8554c158bcd942890729d9dad40074b70b66f"}
Oct 29 22:03:06.360954 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:06.360785    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:03:06.439365 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:06.439294    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n" podStartSLOduration=7.354769469 podStartE2EDuration="13.439275873s" totalImagesPullingTime="6.084506404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:06.422323793 +0000 UTC m=+1002.579449317" watchObservedRunningTime="2026-10-29 22:03:06.439275873 +0000 UTC m=+1002.596401374"
Oct 29 22:03:07.213941 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:07.213906    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:07.372044 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:07.372008    4550 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zlz8n container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:03:07.372245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:07.372076    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n" podUID="34be22aa-35ff-4fb0-90d8-f2a523511916" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:03:09.089913 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:09.089875    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:10.371336 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:10.371309    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zlz8n"
Oct 29 22:03:10.964643 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:10.964603    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:11.211268 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.211234    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"]
Oct 29 22:03:11.215871 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.215808    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.254933 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.254901    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"]
Oct 29 22:03:11.310226 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.310185    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-751a7416-openshift-default-dockercfg-sx494\""
Oct 29 22:03:11.315985 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.315951    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.316077 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.316061    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ce2b92-3933-439a-b273-d313d80ecaf1-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.316120 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.316107    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.316200 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.316187    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxbzm\" (UniqueName: \"kubernetes.io/projected/73ce2b92-3933-439a-b273-d313d80ecaf1-kube-api-access-lxbzm\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.316268 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.316247    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.316307 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.316283    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.316346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.316307    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.316416 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.316403    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.316578 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.316564    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.417687 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.417659    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxbzm\" (UniqueName: \"kubernetes.io/projected/73ce2b92-3933-439a-b273-d313d80ecaf1-kube-api-access-lxbzm\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.418265 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.417713    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.418265 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.417747    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.418265 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.417781    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.418265 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.417837    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.418265 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.418033    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.418541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.418277    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.418541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.418324    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.418541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.418276    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.418541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.418359    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ce2b92-3933-439a-b273-d313d80ecaf1-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.418541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.418398    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.418541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.418403    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.418872 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.418657    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.419016 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.418996    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ce2b92-3933-439a-b273-d313d80ecaf1-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.421068 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.421045    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.421297 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.421279    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.441304 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.441278    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.441526 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.441509    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxbzm\" (UniqueName: \"kubernetes.io/projected/73ce2b92-3933-439a-b273-d313d80ecaf1-kube-api-access-lxbzm\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.491200 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:03:11.491087    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:03:11.534796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.534766    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:11.706221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.706190    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:11.856892 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.856863    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"]
Oct 29 22:03:11.858258 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:03:11.858232    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ce2b92_3933_439a_b273_d313d80ecaf1.slice/crio-250a813c73cc13add8be1c21b319b442acd1d751a0d43f0cf0ac7a7e3147ffc1 WatchSource:0}: Error finding container 250a813c73cc13add8be1c21b319b442acd1d751a0d43f0cf0ac7a7e3147ffc1: Status 404 returned error can't find the container with id 250a813c73cc13add8be1c21b319b442acd1d751a0d43f0cf0ac7a7e3147ffc1
Oct 29 22:03:11.860806 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.860776    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:03:11.860906 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.860854    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:03:11.860906 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:11.860899    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:03:12.406339 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:12.406303    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9" event={"ID":"73ce2b92-3933-439a-b273-d313d80ecaf1","Type":"ContainerStarted","Data":"ef1639f46473a97e7e9800a67deac92528ca8dd3362c8fb309f33096e033ce39"}
Oct 29 22:03:12.406507 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:12.406365    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9" event={"ID":"73ce2b92-3933-439a-b273-d313d80ecaf1","Type":"ContainerStarted","Data":"250a813c73cc13add8be1c21b319b442acd1d751a0d43f0cf0ac7a7e3147ffc1"}
Oct 29 22:03:12.467399 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:12.467343    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9" podStartSLOduration=1.467330338 podStartE2EDuration="1.467330338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:12.466212863 +0000 UTC m=+1008.623338361" watchObservedRunningTime="2026-10-29 22:03:12.467330338 +0000 UTC m=+1008.624455835"
Oct 29 22:03:12.535449 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:12.535414    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:12.536793 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:12.536767    4550 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:03:12.536912 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:12.536817    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9" podUID="73ce2b92-3933-439a-b273-d313d80ecaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 29 22:03:12.742087 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:12.741990    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:12.879171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:12.875764    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:13.460636 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:13.460597    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:13.535956 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:13.535919    4550 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:03:13.536386 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:13.535974    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9" podUID="73ce2b92-3933-439a-b273-d313d80ecaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 29 22:03:13.586130 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:13.586099    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:13.997169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:13.996246    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:14.230166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:14.220541    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:14.536300 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:14.536266    4550 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:03:14.537352 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:14.536325    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9" podUID="73ce2b92-3933-439a-b273-d313d80ecaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 29 22:03:14.749921 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:14.749880    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:15.535897 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:15.535860    4550 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:03:15.536076 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:15.535913    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9" podUID="73ce2b92-3933-439a-b273-d313d80ecaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 29 22:03:16.535500 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:16.535458    4550 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:03:16.536208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:16.535524    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9" podUID="73ce2b92-3933-439a-b273-d313d80ecaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 29 22:03:16.611184 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:16.611116    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:17.535764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:17.535684    4550 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:03:17.536128 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:17.535763    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9" podUID="73ce2b92-3933-439a-b273-d313d80ecaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 29 22:03:18.487202 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:18.487166    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:18.535357 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:18.535303    4550 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 29 22:03:18.535742 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:18.535705    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9" podUID="73ce2b92-3933-439a-b273-d313d80ecaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 29 22:03:18.829114 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:18.829078    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:18.981421 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:18.981384    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:19.539871 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:19.539842    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:19.540367 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:19.540348    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:19.540927 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:19.540906    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:03:20.479353 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:20.479315    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:20.600693 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:20.600657    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:22.190045 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:22.189997    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:22.318076 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:22.318040    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:24.058094 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:24.058057    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:25.733112 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:25.733075    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:27.518028 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:27.517983    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:29.196195 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:29.196162    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:29.675458 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:29.675426    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"]
Oct 29 22:03:29.675710 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:29.675666    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw" podUID="3deae563-f3ea-41cd-bd22-36ca90375806" containerName="istio-proxy" containerID="cri-o://364b733813fa4052cb66474978426de8deaf019c517bcfd6dde7747b8aa7ddcb" gracePeriod=30
Oct 29 22:03:30.808682 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:30.808655    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:32.418189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:32.418126    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:34.118825 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:34.118791    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:34.958093 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:34.958031    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 22:03:35.044008 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.043978    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-workload-socket\") pod \"3deae563-f3ea-41cd-bd22-36ca90375806\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") "
Oct 29 22:03:35.044230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.044034    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3deae563-f3ea-41cd-bd22-36ca90375806-kube-api-access-64bss\" (UniqueName: \"kubernetes.io/projected/3deae563-f3ea-41cd-bd22-36ca90375806-kube-api-access-64bss\") pod \"3deae563-f3ea-41cd-bd22-36ca90375806\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") "
Oct 29 22:03:35.044230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.044081    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-istio-data\") pod \"3deae563-f3ea-41cd-bd22-36ca90375806\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") "
Oct 29 22:03:35.044230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.044132    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-workload-certs\") pod \"3deae563-f3ea-41cd-bd22-36ca90375806\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") "
Oct 29 22:03:35.044444 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.044237    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-istio-envoy\") pod \"3deae563-f3ea-41cd-bd22-36ca90375806\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") "
Oct 29 22:03:35.044444 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.044309    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3deae563-f3ea-41cd-bd22-36ca90375806-istio-token\" (UniqueName: \"kubernetes.io/projected/3deae563-f3ea-41cd-bd22-36ca90375806-istio-token\") pod \"3deae563-f3ea-41cd-bd22-36ca90375806\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") "
Oct 29 22:03:35.044444 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.044323    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-workload-socket" pod "3deae563-f3ea-41cd-bd22-36ca90375806" (UID: "3deae563-f3ea-41cd-bd22-36ca90375806"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:03:35.044444 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.044383    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3deae563-f3ea-41cd-bd22-36ca90375806-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3deae563-f3ea-41cd-bd22-36ca90375806-istiod-ca-cert\") pod \"3deae563-f3ea-41cd-bd22-36ca90375806\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") "
Oct 29 22:03:35.044664 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.044436    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-istio-data" pod "3deae563-f3ea-41cd-bd22-36ca90375806" (UID: "3deae563-f3ea-41cd-bd22-36ca90375806"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:03:35.044664 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.044464    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-credential-socket\") pod \"3deae563-f3ea-41cd-bd22-36ca90375806\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") "
Oct 29 22:03:35.044664 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.044620    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3deae563-f3ea-41cd-bd22-36ca90375806-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3deae563-f3ea-41cd-bd22-36ca90375806-istio-podinfo\") pod \"3deae563-f3ea-41cd-bd22-36ca90375806\" (UID: \"3deae563-f3ea-41cd-bd22-36ca90375806\") "
Oct 29 22:03:35.044824 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.044764    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-credential-socket" pod "3deae563-f3ea-41cd-bd22-36ca90375806" (UID: "3deae563-f3ea-41cd-bd22-36ca90375806"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:03:35.044824 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.044797    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3deae563-f3ea-41cd-bd22-36ca90375806-istiod-ca-cert" pod "3deae563-f3ea-41cd-bd22-36ca90375806" (UID: "3deae563-f3ea-41cd-bd22-36ca90375806"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:03:35.045214 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.045187    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-workload-certs" pod "3deae563-f3ea-41cd-bd22-36ca90375806" (UID: "3deae563-f3ea-41cd-bd22-36ca90375806"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:03:35.045416 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.045384    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3deae563-f3ea-41cd-bd22-36ca90375806-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:03:35.045491 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.045420    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:03:35.045491 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.045436    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:03:35.045491 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.045449    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:03:35.045491 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.045463    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:03:35.047283 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.047260    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3deae563-f3ea-41cd-bd22-36ca90375806-istio-podinfo" pod "3deae563-f3ea-41cd-bd22-36ca90375806" (UID: "3deae563-f3ea-41cd-bd22-36ca90375806"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:03:35.047557 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.047540    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3deae563-f3ea-41cd-bd22-36ca90375806-istio-token" pod "3deae563-f3ea-41cd-bd22-36ca90375806" (UID: "3deae563-f3ea-41cd-bd22-36ca90375806"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:03:35.049536 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.049511    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3deae563-f3ea-41cd-bd22-36ca90375806-kube-api-access-64bss" pod "3deae563-f3ea-41cd-bd22-36ca90375806" (UID: "3deae563-f3ea-41cd-bd22-36ca90375806"). InnerVolumeSpecName "kube-api-access-64bss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:03:35.050671 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.050648    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-istio-envoy" pod "3deae563-f3ea-41cd-bd22-36ca90375806" (UID: "3deae563-f3ea-41cd-bd22-36ca90375806"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:03:35.146663 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.146628    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3deae563-f3ea-41cd-bd22-36ca90375806-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:03:35.146663 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.146660    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64bss\" (UniqueName: \"kubernetes.io/projected/3deae563-f3ea-41cd-bd22-36ca90375806-kube-api-access-64bss\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:03:35.146663 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.146670    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3deae563-f3ea-41cd-bd22-36ca90375806-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:03:35.147114 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.146681    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3deae563-f3ea-41cd-bd22-36ca90375806-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:03:35.594218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.594181    4550 generic.go:356] "Generic (PLEG): container finished" podID="3deae563-f3ea-41cd-bd22-36ca90375806" containerID="364b733813fa4052cb66474978426de8deaf019c517bcfd6dde7747b8aa7ddcb" exitCode=0
Oct 29 22:03:35.594392 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.594228    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"
Oct 29 22:03:35.594392 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.594257    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw" event={"ID":"3deae563-f3ea-41cd-bd22-36ca90375806","Type":"ContainerDied","Data":"364b733813fa4052cb66474978426de8deaf019c517bcfd6dde7747b8aa7ddcb"}
Oct 29 22:03:35.594392 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.594307    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw" event={"ID":"3deae563-f3ea-41cd-bd22-36ca90375806","Type":"ContainerDied","Data":"a3458f8a5d83dae0ba239155511aa81f3d6c6960ffe2bc0011640a95f89ed31a"}
Oct 29 22:03:35.594392 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.594329    4550 scope.go:122] "RemoveContainer" containerID="364b733813fa4052cb66474978426de8deaf019c517bcfd6dde7747b8aa7ddcb"
Oct 29 22:03:35.604929 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.604911    4550 scope.go:122] "RemoveContainer" containerID="364b733813fa4052cb66474978426de8deaf019c517bcfd6dde7747b8aa7ddcb"
Oct 29 22:03:35.605276 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:03:35.605249    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"364b733813fa4052cb66474978426de8deaf019c517bcfd6dde7747b8aa7ddcb\": container with ID starting with 364b733813fa4052cb66474978426de8deaf019c517bcfd6dde7747b8aa7ddcb not found: ID does not exist" containerID="364b733813fa4052cb66474978426de8deaf019c517bcfd6dde7747b8aa7ddcb"
Oct 29 22:03:35.605371 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.605276    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"364b733813fa4052cb66474978426de8deaf019c517bcfd6dde7747b8aa7ddcb"} err="failed to get container status \"364b733813fa4052cb66474978426de8deaf019c517bcfd6dde7747b8aa7ddcb\": rpc error: code = NotFound desc = could not find container \"364b733813fa4052cb66474978426de8deaf019c517bcfd6dde7747b8aa7ddcb\": container with ID starting with 364b733813fa4052cb66474978426de8deaf019c517bcfd6dde7747b8aa7ddcb not found: ID does not exist"
Oct 29 22:03:35.667278 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.667244    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"]
Oct 29 22:03:35.679222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.679191    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tpxsw"]
Oct 29 22:03:35.739886 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.739848    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:35.870285 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.870133    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:35.994265 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.994232    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3deae563-f3ea-41cd-bd22-36ca90375806" path="/var/lib/kubelet/pods/3deae563-f3ea-41cd-bd22-36ca90375806/volumes"
Oct 29 22:03:35.994692 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:35.994660    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:37.527706 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:37.527617    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:39.116031 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:39.115972    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:40.789746 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:40.789705    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:41.493007 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:03:41.492971    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:03:42.412974 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:42.412936    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:44.117891 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:44.117863    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:45.797632 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:45.797596    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:47.426572 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:47.426539    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:49.003785 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:49.003684    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:50.725094 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:50.725054    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:52.405448 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:52.405415    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:54.129111 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:54.129079    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:55.898476 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:55.898447    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:57.716587 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:57.716552    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:59.332354 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:59.332318    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:03:59.451982 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:03:59.451940    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:01.018811 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:01.018775    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:01.500355 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:01.500325    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:01.970320 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:01.970279    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:02.386057 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:02.386013    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:03.042181 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:03.042133    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:04.714662 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:04.714627    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:06.513987 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:06.513951    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:06.631934 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:06.631901    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:08.172914 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:08.172870    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:08.300007 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:08.299973    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:08.420296 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:08.420263    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:10.149607 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:10.149578    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:11.492257 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:04:11.492220    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:04:11.834463 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:11.834433    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:11.954253 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:11.954224    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:12.077332 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:12.077301    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:13.662070 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:13.662041    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:15.299631 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:15.299601    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:16.912319 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:16.912285    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:18.555814 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:18.555783    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:20.126587 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:20.126540    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:20.865459 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:20.865417    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:20.992308 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:20.992272    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:21.921293 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:21.921255    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:24.520779 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:24.486749    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:24.520779 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:24.516333    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:04:24.632189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:24.617958    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:24.807226 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:24.777483    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:24.915054 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:24.913638    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:24.915054 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:24.913742    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6" podUID="5a89ba55-981b-475e-a86c-833c547e65d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.277348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:25.251220    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.456455 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:25.456357    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.486849 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:25.486810    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.488224 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:25.488196    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:04:25.521321 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:25.521276    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.523284 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:25.521881    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.524021 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:25.524001    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:04:25.551301 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:25.551258    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:04:25.582133 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:25.582092    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:04:25.618083 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:25.611322    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:04:25.629456 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:25.619363    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.629456 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:25.619760    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:04:26.260164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:26.251421    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:26.344735 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:26.336321    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:26.344735 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:26.336434    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:04:26.665161 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:26.654234    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:04:26.665161 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:26.655365    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:04:26.690012 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:26.680353    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Oct 29 22:04:26.690012 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:26.680689    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:26.748425 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:26.691593    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:04:26.748425 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:26.723271    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:04:26.817134 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:26.802473    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:26.893321 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:26.859038    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:27.220168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.197493    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:27.293183 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.277610    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.345217 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.340185    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.450509 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:26.653045    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:04:27.535224 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.526538    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.686178 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.683350    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.686178 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.683670    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:04:27.686178 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.683926    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.686178 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.684123    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:04:27.709515 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.704488    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.718306 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.718258    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.727320 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.727279    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.829222 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:04:27.819898    4550 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.581s"
Oct 29 22:04:27.957423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.957393    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:04:27.957723 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.957704    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:04:27.957993 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.957964    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:04:27.958215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.958200    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:04:27.983169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.907187    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.983466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.959125    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:04:27.983671 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.869247    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.983807 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.893668    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:27.983989 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:27.977283    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.001721 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.001697    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:04:28.002004 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.001987    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:04:28.196326 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.196232    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Oct 29 22:04:28.345278 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.345233    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.676169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.631990    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:04:28.707168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.644403    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.729419 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.729369    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.737263 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.737223    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:04:28.741599 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.741567    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:04:28.763130 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.763100    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:04:28.764329 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.742851    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.764699 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.742884    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:04:28.768162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.742905    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:04:28.768491 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.742930    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:04:28.774537 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.743998    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:04:28.958384 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:28.958286    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.028170 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.003385    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:04:29.047521 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.047436    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.134580 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.134446    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.135616 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.135591    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:04:29.316492 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.303259    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.316492 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.303610    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:04:29.367218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.347060    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:29.367579 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.367533    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.730042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.730005    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.791694 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.750603    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:04:29.792911 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.776800    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:04:29.793365 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.778260    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.793685 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.778303    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:04:29.802159 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.778324    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:04:29.802863 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.778366    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:04:29.816942 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.791935    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.827209 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.827176    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:29.827469 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:29.827436    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf" podUID="db332eb5-5ced-43e9-953b-5d94aa37d706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.050739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.049921    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.093409 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.093364    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.094509 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.094482    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:30.094712 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.094689    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp" podUID="d6233b12-31ad-41d0-bb31-12b2f71161bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.137169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.136868    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.213166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.200410    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:04:30.304786 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.304686    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.384467 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.384422    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.494326 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.494279    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.569233 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.550560    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 29 22:04:30.668289 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.668240    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.816193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.814523    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.840471 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.817844    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:04:30.840471 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.818114    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:04:30.840471 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.826972    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:04:30.840471 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.838219    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:04:30.840471 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.838519    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:04:30.840471 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:30.839615    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:04:31.019918 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.019886    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:31.053307 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.053261    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.101463 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.101356    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:04:31.138541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.138495    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.293489 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.293459    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:04:31.334179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.332259    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.391217 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.389547    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.570469 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.570426    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.815330 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.815231    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.816751 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.816716    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.828266 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.828209    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.828483 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.822217    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.841844 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.841801    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.842769 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.842030    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.884426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.841750    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.925194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:31.888124    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:04:32.106162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:32.105904    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.142201 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:32.139580    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.220165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:32.215218    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:04:32.332785 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:32.332731    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.408250 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:32.406275    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.722368 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:32.722264    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.811321 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:32.811263    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.816275 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:32.816220    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.828362 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:32.828316    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:04:32.829414 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:32.828924    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.829645 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:32.829221    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:04:32.830093 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:32.828989    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:04:32.856166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:32.854438    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.914168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:32.912267    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.919847 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:32.914766    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:04:33.067226 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:33.061007    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded"
Oct 29 22:04:33.119311 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:33.119264    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.145284 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:33.145236    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.170600 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:33.170571    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:04:33.248176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:33.234424    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.341398 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:33.336121    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:04:33.420756 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:33.419751    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.740211 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:33.727759    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:04:33.825167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:33.820674    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.866172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:33.844803    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:04:33.866172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:33.861853    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.866172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:33.863248    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.921184 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:33.912711    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.952616 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:33.940551    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.127172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:34.123241    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.151166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:34.150008    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.421853 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:34.421712    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.685327 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:34.685231    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:04:34.771993 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:34.771941    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:04:34.822735 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:34.822694    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.831244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:34.822957    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:04:34.859263 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:34.859220    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.864270 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:34.864235    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:04:34.865470 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:34.865445    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.915502 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:34.915452    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.941391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:34.941296    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:35.013267 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.002378    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:35.125385 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.125344    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:04:35.151180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.150219    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:35.339793 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.337668    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:04:35.466669 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.466289    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:35.689238 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.688494    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:35.689238 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.689043    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:04:35.779646 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.775241    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:04:35.779646 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.775969    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:04:35.822265 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.808507    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:04:35.853185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.852358    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:04:35.872287 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.872238    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:35.880363 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.872836    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:04:35.913176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.888403    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:04:35.921207 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.894322    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:35.962215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.926772    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:35.997559 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.940489    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:04:36.055009 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:35.946221    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:36.144161 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.138241    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:04:36.167726 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.166598    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:36.346337 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.322481    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:36.347302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.346760    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:36.502165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.501415    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:04:36.527366 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.527328    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:36.646529 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.646443    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:04:36.701198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.697500    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:04:36.725879 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.725846    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:04:36.872104 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.872063    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:04:36.892567 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.892526    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:36.895812 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.895781    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:04:36.896340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.896304    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:36.913564 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.913500    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:04:36.981423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.969777    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:04:36.981423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.969836    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:04:36.981423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.969874    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:04:36.981423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.969907    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:04:36.981423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.969939    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:04:36.995261 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.995231    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:04:36.997289 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:36.997264    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:04:37.565038 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:37.565001    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:39.254655 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:39.254619    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:41.052705 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.052668    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:41.343089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.343046    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:41.492483 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:04:41.492449    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:04:41.666524 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.666431    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"]
Oct 29 22:04:41.667393 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.667366    4550 cpu_manager.go:414] "containerMap: removing container" podUID="3deae563-f3ea-41cd-bd22-36ca90375806" containerName="istio-proxy"
Oct 29 22:04:41.667393 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.667389    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3deae563-f3ea-41cd-bd22-36ca90375806" containerName="istio-proxy"
Oct 29 22:04:41.667641 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.667603    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="3deae563-f3ea-41cd-bd22-36ca90375806" containerName="istio-proxy"
Oct 29 22:04:41.673036 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.673016    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.715711 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.715669    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"]
Oct 29 22:04:41.790134 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.790098    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7cb54e0-6733-4774-be16-ccefc78e7d48-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.790315 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.790183    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m6qw\" (UniqueName: \"kubernetes.io/projected/e7cb54e0-6733-4774-be16-ccefc78e7d48-kube-api-access-2m6qw\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.790315 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.790228    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.790393 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.790322    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.790393 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.790360    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.790460 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.790395    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.790460 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.790450    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.790565 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.790545    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.790618 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.790603    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.828359 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.826032    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:41.891926 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.891895    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7cb54e0-6733-4774-be16-ccefc78e7d48-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.892110 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.891932    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m6qw\" (UniqueName: \"kubernetes.io/projected/e7cb54e0-6733-4774-be16-ccefc78e7d48-kube-api-access-2m6qw\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.892110 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.891964    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.892110 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.891994    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.892348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.892111    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.892348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.892193    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.892348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.892260    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.892348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.892337    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.892620 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.892405    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.892620 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.892411    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.892620 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.892594    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.892793 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.892615    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.892793 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.892684    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.892793 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.892735    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7cb54e0-6733-4774-be16-ccefc78e7d48-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.894905 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.894883    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.895693 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.895669    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.925208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.925101    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m6qw\" (UniqueName: \"kubernetes.io/projected/e7cb54e0-6733-4774-be16-ccefc78e7d48-kube-api-access-2m6qw\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.935390 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.935358    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:41.999668 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:41.999628    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:42.101545 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:42.099444    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:42.373369 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:42.373337    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"]
Oct 29 22:04:42.374473 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:04:42.374445    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cb54e0_6733_4774_be16_ccefc78e7d48.slice/crio-91faa73ca86cf6ca7cfb0253169b523a92c4f2a9fc2433b9997aa3fa7c23a583 WatchSource:0}: Error finding container 91faa73ca86cf6ca7cfb0253169b523a92c4f2a9fc2433b9997aa3fa7c23a583: Status 404 returned error can't find the container with id 91faa73ca86cf6ca7cfb0253169b523a92c4f2a9fc2433b9997aa3fa7c23a583
Oct 29 22:04:42.377589 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:42.377549    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:04:42.377694 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:42.377638    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:04:42.377694 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:42.377684    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:04:42.591164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:42.590379    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc" event={"ID":"e7cb54e0-6733-4774-be16-ccefc78e7d48","Type":"ContainerStarted","Data":"fd1c9b275988d036a78374eed25b008a0677694239c18c44c3b7384b85713e4b"}
Oct 29 22:04:42.591164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:42.590494    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc" event={"ID":"e7cb54e0-6733-4774-be16-ccefc78e7d48","Type":"ContainerStarted","Data":"91faa73ca86cf6ca7cfb0253169b523a92c4f2a9fc2433b9997aa3fa7c23a583"}
Oct 29 22:04:42.689376 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:42.689257    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc" podStartSLOduration=1.689237104 podStartE2EDuration="1.689237104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:42.684320717 +0000 UTC m=+1098.841446230" watchObservedRunningTime="2026-10-29 22:04:42.689237104 +0000 UTC m=+1098.846362603"
Oct 29 22:04:42.724334 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:42.722536    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:43.002484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:43.002391    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:43.003224 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:43.002673    4550 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:04:43.003224 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:43.002861    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc" podUID="e7cb54e0-6733-4774-be16-ccefc78e7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 29 22:04:43.190787 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:43.190749    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:43.337701 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:43.337669    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:43.513264 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:43.510829    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:43.740218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:43.739404    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:43.926077 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:43.926040    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:44.000591 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:44.000489    4550 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:04:44.000796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:44.000591    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc" podUID="e7cb54e0-6733-4774-be16-ccefc78e7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 29 22:04:44.742857 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:44.742813    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:45.003640 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:45.003556    4550 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:04:45.003640 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:45.003616    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc" podUID="e7cb54e0-6733-4774-be16-ccefc78e7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 29 22:04:46.001037 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:46.001003    4550 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:04:46.001435 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:46.001056    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc" podUID="e7cb54e0-6733-4774-be16-ccefc78e7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 29 22:04:46.989617 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:46.989577    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:47.000449 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:47.000422    4550 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:04:47.000586 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:47.000466    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc" podUID="e7cb54e0-6733-4774-be16-ccefc78e7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 29 22:04:47.585680 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:47.585646    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:47.716075 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:47.716043    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:48.001009 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:48.000931    4550 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:04:48.001009 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:48.000980    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc" podUID="e7cb54e0-6733-4774-be16-ccefc78e7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 29 22:04:49.000896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:49.000854    4550 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 29 22:04:49.001303 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:49.000920    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc" podUID="e7cb54e0-6733-4774-be16-ccefc78e7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 29 22:04:49.013149 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:49.013104    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:49.416928 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:49.416899    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:49.539324 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:49.539292    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:50.136645 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:50.136600    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:50.137603 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:50.137571    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:50.138248 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:50.138226    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:04:51.016905 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:51.016878    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:52.724774 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:52.724715    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:54.293545 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:54.293515    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:54.591134 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:54.591106    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:54.917846 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:54.917768    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:55.759792 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:55.759759    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"]
Oct 29 22:04:55.760285 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:55.760001    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb" podUID="24ec2d58-3402-4880-86e8-ab9fdbc4a9c8" containerName="maas-api" containerID="cri-o://ea55eec8903227761b24567c708a0bb8540d6335c26c277812c77251796c04e9" gracePeriod=30
Oct 29 22:04:55.883207 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:55.883119    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:56.555015 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.554983    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:04:56.748438 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.748401    4550 generic.go:356] "Generic (PLEG): container finished" podID="24ec2d58-3402-4880-86e8-ab9fdbc4a9c8" containerID="ea55eec8903227761b24567c708a0bb8540d6335c26c277812c77251796c04e9" exitCode=0
Oct 29 22:04:56.748648 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.748483    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb" event={"ID":"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8","Type":"ContainerDied","Data":"ea55eec8903227761b24567c708a0bb8540d6335c26c277812c77251796c04e9"}
Oct 29 22:04:56.748648 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.748572    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb" event={"ID":"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8","Type":"ContainerDied","Data":"99202135ddaf32eca3a91695114deddae670142eb2299edb180bf05dfe0831dc"}
Oct 29 22:04:56.748648 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.748606    4550 scope.go:122] "RemoveContainer" containerID="ea55eec8903227761b24567c708a0bb8540d6335c26c277812c77251796c04e9"
Oct 29 22:04:56.748901 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.748882    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"
Oct 29 22:04:56.750967 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.750945    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-openshift-service-ca\") pod \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") "
Oct 29 22:04:56.751454 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.751433    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-metrics-certs\") pod \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") "
Oct 29 22:04:56.751564 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.751504    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-kube-api-access-zvqcn\" (UniqueName: \"kubernetes.io/projected/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-kube-api-access-zvqcn\") pod \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") "
Oct 29 22:04:56.752223 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.751928    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-openshift-service-ca" pod "24ec2d58-3402-4880-86e8-ab9fdbc4a9c8" (UID: "24ec2d58-3402-4880-86e8-ab9fdbc4a9c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:04:56.754300 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.754255    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-maas-api-tls\") pod \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\" (UID: \"24ec2d58-3402-4880-86e8-ab9fdbc4a9c8\") "
Oct 29 22:04:56.755035 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.755009    4550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-openshift-service-ca\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:04:56.756003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.755980    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-kube-api-access-zvqcn" pod "24ec2d58-3402-4880-86e8-ab9fdbc4a9c8" (UID: "24ec2d58-3402-4880-86e8-ab9fdbc4a9c8"). InnerVolumeSpecName "kube-api-access-zvqcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:04:56.758032 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.757975    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-maas-api-tls" pod "24ec2d58-3402-4880-86e8-ab9fdbc4a9c8" (UID: "24ec2d58-3402-4880-86e8-ab9fdbc4a9c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:04:56.758585 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.758555    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-metrics-certs" pod "24ec2d58-3402-4880-86e8-ab9fdbc4a9c8" (UID: "24ec2d58-3402-4880-86e8-ab9fdbc4a9c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:04:56.765923 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.765813    4550 scope.go:122] "RemoveContainer" containerID="ea55eec8903227761b24567c708a0bb8540d6335c26c277812c77251796c04e9"
Oct 29 22:04:56.766238 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.765959    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:56.766298 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:04:56.766269    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea55eec8903227761b24567c708a0bb8540d6335c26c277812c77251796c04e9\": container with ID starting with ea55eec8903227761b24567c708a0bb8540d6335c26c277812c77251796c04e9 not found: ID does not exist" containerID="ea55eec8903227761b24567c708a0bb8540d6335c26c277812c77251796c04e9"
Oct 29 22:04:56.766352 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.766309    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea55eec8903227761b24567c708a0bb8540d6335c26c277812c77251796c04e9"} err="failed to get container status \"ea55eec8903227761b24567c708a0bb8540d6335c26c277812c77251796c04e9\": rpc error: code = NotFound desc = could not find container \"ea55eec8903227761b24567c708a0bb8540d6335c26c277812c77251796c04e9\": container with ID starting with ea55eec8903227761b24567c708a0bb8540d6335c26c277812c77251796c04e9 not found: ID does not exist"
Oct 29 22:04:56.856559 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.856471    4550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-metrics-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:04:56.856559 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.856508    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvqcn\" (UniqueName: \"kubernetes.io/projected/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-kube-api-access-zvqcn\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:04:56.856559 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.856523    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8-maas-api-tls\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:04:56.948648 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:56.948603    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:57.070604 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:57.070519    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:57.109885 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:57.109844    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"]
Oct 29 22:04:57.119346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:57.119316    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a9f9aa-8495ccd844-l49wb"]
Oct 29 22:04:57.496059 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:57.496021    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:57.774640 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:57.774547    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:57.990305 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:57.990276    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ec2d58-3402-4880-86e8-ab9fdbc4a9c8" path="/var/lib/kubelet/pods/24ec2d58-3402-4880-86e8-ab9fdbc4a9c8/volumes"
Oct 29 22:04:58.135514 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:58.135483    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:58.263331 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:58.263298    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:04:58.855160 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:04:58.849940    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:00.627239 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:00.627205    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:02.236839 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:02.236810    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:03.956301 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:03.956263    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:04.085329 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:04.085240    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:05.057200 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:05.057165    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:05.181420 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:05.177303    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:05.450500 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:05.450475    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:05.758279 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:05.758191    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:07.883178 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:07.883077    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:08.479199 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:08.479130    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:08.993585 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:08.993553    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:09.151213 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:09.147589    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:09.884854 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:09.884816    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:10.028874 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:10.028841    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:10.253440 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:10.253352    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:11.493906 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:05:11.493867    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:05:12.047906 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:12.047875    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:12.171064 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:12.171020    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:14.016693 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:14.016658    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:14.138504 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:14.138473    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:14.313748 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:14.313662    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:15.035529 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:15.035500    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"]
Oct 29 22:05:15.047350 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:15.047324    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e4503e9-dcnd5"]
Oct 29 22:05:15.667522 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:15.667489    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:15.791391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:15.791360    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:16.169303 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:16.169246    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:16.181071 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:16.181036    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c512471-7b82-40c8-967f-8b96d57ac572" path="/var/lib/kubelet/pods/3c512471-7b82-40c8-967f-8b96d57ac572/volumes"
Oct 29 22:05:16.299653 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:16.299612    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:16.646827 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:16.646794    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:16.771567 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:16.771532    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:17.845707 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:17.845663    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:19.762732 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:19.762693    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:20.987692 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:20.987648    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:21.160193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:21.156664    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:22.120623 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:22.120593    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:23.603014 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:23.602971    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:23.813836 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:23.813794    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:25.579929 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:25.579885    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:27.387930 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:27.387897    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:29.375693 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:29.375659    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:31.381622 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:31.381591    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:32.009696 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:32.009666    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:32.707426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:32.707372    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:33.240388 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:33.240347    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:33.390567 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:33.390535    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:33.682248 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:33.682197    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:35.515254 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:35.515182    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:37.123646 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:37.123617    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:37.286664 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:37.286630    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:37.486753 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:37.486661    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:37.670253 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:37.670211    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:38.560899 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:38.560852    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:39.183698 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.183664    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:39.416284 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.416246    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"]
Oct 29 22:05:39.417213 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.417187    4550 cpu_manager.go:414] "containerMap: removing container" podUID="24ec2d58-3402-4880-86e8-ab9fdbc4a9c8" containerName="maas-api"
Oct 29 22:05:39.417404 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.417382    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ec2d58-3402-4880-86e8-ab9fdbc4a9c8" containerName="maas-api"
Oct 29 22:05:39.417701 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.417682    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ec2d58-3402-4880-86e8-ab9fdbc4a9c8" containerName="maas-api"
Oct 29 22:05:39.424315 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.424289    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"
Oct 29 22:05:39.426714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.426687    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:39.458730 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.458663    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a26f29\""
Oct 29 22:05:39.491921 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.491879    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"]
Oct 29 22:05:39.505729 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.505697    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"]
Oct 29 22:05:39.511062 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.511033    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"
Oct 29 22:05:39.528971 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.528939    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a26f29-dockercfg-njd9x\""
Oct 29 22:05:39.541442 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.541410    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"]
Oct 29 22:05:39.557407 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.557381    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:39.572432 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.572396    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxszn\" (UniqueName: \"kubernetes.io/projected/f061ec29-6d39-43d9-93ea-de6efb4a6489-kube-api-access-qxszn\") pod \"payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq\" (UID: \"f061ec29-6d39-43d9-93ea-de6efb4a6489\") " pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"
Oct 29 22:05:39.572832 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.572689    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f061ec29-6d39-43d9-93ea-de6efb4a6489-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq\" (UID: \"f061ec29-6d39-43d9-93ea-de6efb4a6489\") " pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"
Oct 29 22:05:39.675187 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.675065    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m7v2\" (UniqueName: \"kubernetes.io/projected/2625bcf3-456a-4112-9c40-d42f7da5cfda-kube-api-access-5m7v2\") pod \"payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp\" (UID: \"2625bcf3-456a-4112-9c40-d42f7da5cfda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"
Oct 29 22:05:39.675187 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.675124    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxszn\" (UniqueName: \"kubernetes.io/projected/f061ec29-6d39-43d9-93ea-de6efb4a6489-kube-api-access-qxszn\") pod \"payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq\" (UID: \"f061ec29-6d39-43d9-93ea-de6efb4a6489\") " pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"
Oct 29 22:05:39.675445 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.675280    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2625bcf3-456a-4112-9c40-d42f7da5cfda-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp\" (UID: \"2625bcf3-456a-4112-9c40-d42f7da5cfda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"
Oct 29 22:05:39.675507 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.675443    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f061ec29-6d39-43d9-93ea-de6efb4a6489-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq\" (UID: \"f061ec29-6d39-43d9-93ea-de6efb4a6489\") " pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"
Oct 29 22:05:39.703414 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.703118    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f061ec29-6d39-43d9-93ea-de6efb4a6489-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq\" (UID: \"f061ec29-6d39-43d9-93ea-de6efb4a6489\") " pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"
Oct 29 22:05:39.742509 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.738410    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxszn\" (UniqueName: \"kubernetes.io/projected/f061ec29-6d39-43d9-93ea-de6efb4a6489-kube-api-access-qxszn\") pod \"payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq\" (UID: \"f061ec29-6d39-43d9-93ea-de6efb4a6489\") " pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"
Oct 29 22:05:39.747169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.744411    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"
Oct 29 22:05:39.809163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.807776    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m7v2\" (UniqueName: \"kubernetes.io/projected/2625bcf3-456a-4112-9c40-d42f7da5cfda-kube-api-access-5m7v2\") pod \"payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp\" (UID: \"2625bcf3-456a-4112-9c40-d42f7da5cfda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"
Oct 29 22:05:39.809163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.808695    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2625bcf3-456a-4112-9c40-d42f7da5cfda-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp\" (UID: \"2625bcf3-456a-4112-9c40-d42f7da5cfda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"
Oct 29 22:05:39.815748 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.815721    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2625bcf3-456a-4112-9c40-d42f7da5cfda-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp\" (UID: \"2625bcf3-456a-4112-9c40-d42f7da5cfda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"
Oct 29 22:05:39.853196 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:39.853165    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m7v2\" (UniqueName: \"kubernetes.io/projected/2625bcf3-456a-4112-9c40-d42f7da5cfda-kube-api-access-5m7v2\") pod \"payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp\" (UID: \"2625bcf3-456a-4112-9c40-d42f7da5cfda\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"
Oct 29 22:05:40.084237 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.083494    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:40.154162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.147695    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"
Oct 29 22:05:40.410191 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.406571    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"]
Oct 29 22:05:40.428470 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.423274    4550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:05:40.432734 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.432711    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:40.604959 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.604837    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:40.619353 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.618899    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"]
Oct 29 22:05:40.624889 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.624865    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.669413 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.667882    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2d53cd-openshift-default-dockercfg-2rvtf\""
Oct 29 22:05:40.669413 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.667993    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"]
Oct 29 22:05:40.723606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.723562    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"]
Oct 29 22:05:40.725462 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.725423    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-data\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.725653 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.725631    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-workload-socket\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.725902 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.725873    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/097cc026-11a8-45b2-8eda-26fce58e7a02-istiod-ca-cert\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.726063 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.726050    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-envoy\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.726237 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.726217    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9bxn\" (UniqueName: \"kubernetes.io/projected/097cc026-11a8-45b2-8eda-26fce58e7a02-kube-api-access-h9bxn\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.726426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.726406    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-credential-socket\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.730837 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.730812    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-workload-certs\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.730998 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.730981    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-token\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.732556 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.732514    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-podinfo\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.762886 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.759094    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:40.977222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.966638    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/097cc026-11a8-45b2-8eda-26fce58e7a02-istiod-ca-cert\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.977516 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.977487    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-envoy\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.977756 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.977737    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9bxn\" (UniqueName: \"kubernetes.io/projected/097cc026-11a8-45b2-8eda-26fce58e7a02-kube-api-access-h9bxn\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.977905 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.977889    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-credential-socket\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.978104 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.978088    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-workload-certs\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.978255 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.978240    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-token\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.978413 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.978398    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-podinfo\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.986347 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.986316    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-data\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.986561 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.986538    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-workload-socket\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.998177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.987111    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-workload-socket\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.998384 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.987657    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/097cc026-11a8-45b2-8eda-26fce58e7a02-istiod-ca-cert\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.998502 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.988057    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-data\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.998595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.979200    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-workload-certs\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:40.998678 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:40.979449    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-credential-socket\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:41.050552 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:41.045217    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-token\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:41.080728 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:41.080658    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-envoy\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:41.081576 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:41.081548    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-podinfo\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:41.084857 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:41.084799    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9bxn\" (UniqueName: \"kubernetes.io/projected/097cc026-11a8-45b2-8eda-26fce58e7a02-kube-api-access-h9bxn\") pod \"e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") " pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:41.193349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:41.193314    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:41.248678 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:41.248245    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:41.413171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:41.400690    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq" event={"ID":"f061ec29-6d39-43d9-93ea-de6efb4a6489","Type":"ContainerStarted","Data":"dcd7d910501255e3fe765e7a0e14656c25f137ba38e8ed9c76314b4095a777d2"}
Oct 29 22:05:41.427668 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:41.427579    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp" event={"ID":"2625bcf3-456a-4112-9c40-d42f7da5cfda","Type":"ContainerStarted","Data":"9433bc09be6627e0ea38a8cfaa12445f7323f8210e54d7c14dc3b29347269ce6"}
Oct 29 22:05:41.567670 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:05:41.567546    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:05:41.637733 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:41.637696    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"]
Oct 29 22:05:41.640762 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:05:41.640728    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097cc026_11a8_45b2_8eda_26fce58e7a02.slice/crio-d00a5269cea9969865f1c90a6a25f8bbe60b0bce05f8b97b9a19819b5a12ffb1 WatchSource:0}: Error finding container d00a5269cea9969865f1c90a6a25f8bbe60b0bce05f8b97b9a19819b5a12ffb1: Status 404 returned error can't find the container with id d00a5269cea9969865f1c90a6a25f8bbe60b0bce05f8b97b9a19819b5a12ffb1
Oct 29 22:05:41.646093 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:41.645490    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:05:41.646093 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:41.645566    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:05:41.646093 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:41.645594    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:05:41.771368 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:41.771333    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:42.449561 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:42.449517    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w" event={"ID":"097cc026-11a8-45b2-8eda-26fce58e7a02","Type":"ContainerStarted","Data":"a2746305a38ab0891c511dd8ef7e79ff52f59c93741d58dc873b85ca742988bb"}
Oct 29 22:05:42.449720 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:42.449630    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w" event={"ID":"097cc026-11a8-45b2-8eda-26fce58e7a02","Type":"ContainerStarted","Data":"d00a5269cea9969865f1c90a6a25f8bbe60b0bce05f8b97b9a19819b5a12ffb1"}
Oct 29 22:05:42.509984 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:42.509927    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w" podStartSLOduration=2.509910625 podStartE2EDuration="2.509910625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:42.509082664 +0000 UTC m=+1158.666208162" watchObservedRunningTime="2026-10-29 22:05:42.509910625 +0000 UTC m=+1158.667036125"
Oct 29 22:05:43.249839 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:43.249413    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:43.296188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:43.295482    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:43.458035 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:43.457968    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:43.459346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:43.459307    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:05:43.641051 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:43.641018    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:44.468089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:44.467950    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp" event={"ID":"2625bcf3-456a-4112-9c40-d42f7da5cfda","Type":"ContainerStarted","Data":"7f3131fd0479d0b49f5d2d50039bb0ce6a3d58009c2177a103a68401342ab7bd"}
Oct 29 22:05:44.468089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:44.468039    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"
Oct 29 22:05:44.469907 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:44.469882    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq" event={"ID":"f061ec29-6d39-43d9-93ea-de6efb4a6489","Type":"ContainerStarted","Data":"f78516fa9673215b6a174da2a012ac130f30c890196675b8f9928f6806cb664d"}
Oct 29 22:05:44.470102 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:44.470082    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"
Oct 29 22:05:44.510074 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:44.510030    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp" podStartSLOduration=1.981469076 podStartE2EDuration="5.51001735s" totalImagesPullingTime="3.528548274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:44.509529747 +0000 UTC m=+1160.666655246" watchObservedRunningTime="2026-10-29 22:05:44.51001735 +0000 UTC m=+1160.667142848"
Oct 29 22:05:44.556977 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:44.556877    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq" podStartSLOduration=1.7000471849999998 podStartE2EDuration="5.556858447s" totalImagesPullingTime="3.856811262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:44.551057435 +0000 UTC m=+1160.708182941" watchObservedRunningTime="2026-10-29 22:05:44.556858447 +0000 UTC m=+1160.713983946"
Oct 29 22:05:44.990568 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:44.990537    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:45.119279 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:45.119235    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:45.739938 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:45.739907    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:46.037525 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:46.025467    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:46.751430 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:46.751386    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:46.852248 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:46.852219    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"]
Oct 29 22:05:46.856710 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:46.856690    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"
Oct 29 22:05:46.882730 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:46.882698    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"]
Oct 29 22:05:46.885881 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:46.885435    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2d53cd-dockercfg-pchh6\""
Oct 29 22:05:46.896054 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:46.896028    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:46.993446 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:46.986436    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2d53cd\""
Oct 29 22:05:47.038227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:47.037987    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db60d8d1-7c00-47f5-bff6-a70d7456a389-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs\" (UID: \"db60d8d1-7c00-47f5-bff6-a70d7456a389\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"
Oct 29 22:05:47.038227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:47.038066    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j4lp\" (UniqueName: \"kubernetes.io/projected/db60d8d1-7c00-47f5-bff6-a70d7456a389-kube-api-access-9j4lp\") pod \"payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs\" (UID: \"db60d8d1-7c00-47f5-bff6-a70d7456a389\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"
Oct 29 22:05:47.144532 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:47.144484    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db60d8d1-7c00-47f5-bff6-a70d7456a389-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs\" (UID: \"db60d8d1-7c00-47f5-bff6-a70d7456a389\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"
Oct 29 22:05:47.148183 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:47.148155    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j4lp\" (UniqueName: \"kubernetes.io/projected/db60d8d1-7c00-47f5-bff6-a70d7456a389-kube-api-access-9j4lp\") pod \"payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs\" (UID: \"db60d8d1-7c00-47f5-bff6-a70d7456a389\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"
Oct 29 22:05:47.153229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:47.146872    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db60d8d1-7c00-47f5-bff6-a70d7456a389-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs\" (UID: \"db60d8d1-7c00-47f5-bff6-a70d7456a389\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"
Oct 29 22:05:47.191172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:47.190963    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j4lp\" (UniqueName: \"kubernetes.io/projected/db60d8d1-7c00-47f5-bff6-a70d7456a389-kube-api-access-9j4lp\") pod \"payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs\" (UID: \"db60d8d1-7c00-47f5-bff6-a70d7456a389\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"
Oct 29 22:05:47.479372 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:47.479336    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"
Oct 29 22:05:47.813891 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:47.813865    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"]
Oct 29 22:05:47.815836 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:05:47.815809    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb60d8d1_7c00_47f5_bff6_a70d7456a389.slice/crio-d2fc62392a35321fd7e8296e49a777aeb5d78b60d2b22b2e0e69201bedccaa28 WatchSource:0}: Error finding container d2fc62392a35321fd7e8296e49a777aeb5d78b60d2b22b2e0e69201bedccaa28: Status 404 returned error can't find the container with id d2fc62392a35321fd7e8296e49a777aeb5d78b60d2b22b2e0e69201bedccaa28
Oct 29 22:05:48.282163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:48.278488    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:48.507350 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:48.507318    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:48.510661 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:48.510628    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs" event={"ID":"db60d8d1-7c00-47f5-bff6-a70d7456a389","Type":"ContainerStarted","Data":"6555c9aeb9263beca1a77426fcde9e5950f53a6914fb70eba01c05e606220035"}
Oct 29 22:05:48.510800 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:48.510697    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"
Oct 29 22:05:48.510800 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:48.510724    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs" event={"ID":"db60d8d1-7c00-47f5-bff6-a70d7456a389","Type":"ContainerStarted","Data":"d2fc62392a35321fd7e8296e49a777aeb5d78b60d2b22b2e0e69201bedccaa28"}
Oct 29 22:05:48.556045 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:48.555933    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs" podStartSLOduration=2.555918553 podStartE2EDuration="2.555918553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:48.553101234 +0000 UTC m=+1164.710226769" watchObservedRunningTime="2026-10-29 22:05:48.555918553 +0000 UTC m=+1164.713044050"
Oct 29 22:05:49.036976 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:49.036938    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:49.154356 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:49.154301    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:49.215762 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:49.215727    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"]
Oct 29 22:05:49.221345 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:49.221303    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"
Oct 29 22:05:49.237219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:49.237188    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"]
Oct 29 22:05:49.290681 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:49.290588    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-301fcc-dockercfg-txkh5\""
Oct 29 22:05:49.307557 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:49.299353    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-301fcc\""
Oct 29 22:05:49.422163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:49.414257    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2q2r\" (UniqueName: \"kubernetes.io/projected/5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4-kube-api-access-t2q2r\") pod \"payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw\" (UID: \"5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"
Oct 29 22:05:49.422163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:49.414309    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw\" (UID: \"5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"
Oct 29 22:05:49.516247 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:49.516201    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2q2r\" (UniqueName: \"kubernetes.io/projected/5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4-kube-api-access-t2q2r\") pod \"payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw\" (UID: \"5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"
Oct 29 22:05:49.516431 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:49.516287    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw\" (UID: \"5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"
Oct 29 22:05:49.517680 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:49.517654    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw\" (UID: \"5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"
Oct 29 22:05:49.540754 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:49.540684    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2q2r\" (UniqueName: \"kubernetes.io/projected/5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4-kube-api-access-t2q2r\") pod \"payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw\" (UID: \"5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"
Oct 29 22:05:49.840394 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:49.840367    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"
Oct 29 22:05:50.151925 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:50.151899    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"]
Oct 29 22:05:50.152388 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:05:50.152252    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2f3c23_25b5_4a2d_b44b_765cfb4e51d4.slice/crio-af7add2acebad0f5f2cb645be99fa9b91dc1884db0702a1302827cb94b8473ec WatchSource:0}: Error finding container af7add2acebad0f5f2cb645be99fa9b91dc1884db0702a1302827cb94b8473ec: Status 404 returned error can't find the container with id af7add2acebad0f5f2cb645be99fa9b91dc1884db0702a1302827cb94b8473ec
Oct 29 22:05:50.342412 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:50.342372    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:50.531219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:50.531096    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw" event={"ID":"5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4","Type":"ContainerStarted","Data":"1562b28ea0328fe3cd4b46a22fc15ec5eb856c72a6ad5d4c41574c8dab88cdc7"}
Oct 29 22:05:50.531219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:50.531209    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw" event={"ID":"5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4","Type":"ContainerStarted","Data":"af7add2acebad0f5f2cb645be99fa9b91dc1884db0702a1302827cb94b8473ec"}
Oct 29 22:05:50.531526 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:50.531503    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"
Oct 29 22:05:50.574363 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:50.574311    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw" podStartSLOduration=1.5742983289999999 podStartE2EDuration="1.574298329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:50.572568374 +0000 UTC m=+1166.729693898" watchObservedRunningTime="2026-10-29 22:05:50.574298329 +0000 UTC m=+1166.731423827"
Oct 29 22:05:52.104780 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:52.104743    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:53.799179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:53.799130    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:54.634249 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.634218    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"]
Oct 29 22:05:54.638723 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.638706    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.664847 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.664803    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"]
Oct 29 22:05:54.710426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.710390    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64621438-8bc0-40ce-8fe5-1129114a5fbf-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.710600 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.710467    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64621438-8bc0-40ce-8fe5-1129114a5fbf-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.710600 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.710501    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g5nc\" (UniqueName: \"kubernetes.io/projected/64621438-8bc0-40ce-8fe5-1129114a5fbf-kube-api-access-2g5nc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.710600 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.710594    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64621438-8bc0-40ce-8fe5-1129114a5fbf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.710792 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.710740    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64621438-8bc0-40ce-8fe5-1129114a5fbf-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.710792 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.710764    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64621438-8bc0-40ce-8fe5-1129114a5fbf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.710792 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.710783    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64621438-8bc0-40ce-8fe5-1129114a5fbf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.812722 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.812685    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64621438-8bc0-40ce-8fe5-1129114a5fbf-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.813185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.812733    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64621438-8bc0-40ce-8fe5-1129114a5fbf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.813185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.812766    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64621438-8bc0-40ce-8fe5-1129114a5fbf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.813185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.812871    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64621438-8bc0-40ce-8fe5-1129114a5fbf-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.813185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.812940    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64621438-8bc0-40ce-8fe5-1129114a5fbf-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.813185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.813054    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g5nc\" (UniqueName: \"kubernetes.io/projected/64621438-8bc0-40ce-8fe5-1129114a5fbf-kube-api-access-2g5nc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.813470 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.813211    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64621438-8bc0-40ce-8fe5-1129114a5fbf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.815747 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.815723    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64621438-8bc0-40ce-8fe5-1129114a5fbf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.815855 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.815801    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64621438-8bc0-40ce-8fe5-1129114a5fbf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.815855 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.815800    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64621438-8bc0-40ce-8fe5-1129114a5fbf-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.815934 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.815881    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64621438-8bc0-40ce-8fe5-1129114a5fbf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.816290 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.816272    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64621438-8bc0-40ce-8fe5-1129114a5fbf-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.840874 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.840853    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64621438-8bc0-40ce-8fe5-1129114a5fbf-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.847971 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.847948    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g5nc\" (UniqueName: \"kubernetes.io/projected/64621438-8bc0-40ce-8fe5-1129114a5fbf-kube-api-access-2g5nc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrfmq\" (UID: \"64621438-8bc0-40ce-8fe5-1129114a5fbf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:54.951366 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:54.951297    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:55.282018 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:05:55.281995    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64621438_8bc0_40ce_8fe5_1129114a5fbf.slice/crio-6496daf88793eb10e94b92f4d562a0b7c5afa0fd0acca58625d1399c8612fca6 WatchSource:0}: Error finding container 6496daf88793eb10e94b92f4d562a0b7c5afa0fd0acca58625d1399c8612fca6: Status 404 returned error can't find the container with id 6496daf88793eb10e94b92f4d562a0b7c5afa0fd0acca58625d1399c8612fca6
Oct 29 22:05:55.282732 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:55.282709    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"]
Oct 29 22:05:55.284764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:55.284733    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:05:55.284845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:55.284792    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:05:55.511571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:55.510602    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"
Oct 29 22:05:55.513367 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:55.513329    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"
Oct 29 22:05:55.530320 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:55.530230    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:55.575985 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:55.575945    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq" event={"ID":"64621438-8bc0-40ce-8fe5-1129114a5fbf","Type":"ContainerStarted","Data":"cea173f84a937b95b976461cf2a05030370d907b989ce58ab9167f07c408b64a"}
Oct 29 22:05:55.575985 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:55.575999    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq" event={"ID":"64621438-8bc0-40ce-8fe5-1129114a5fbf","Type":"ContainerStarted","Data":"6496daf88793eb10e94b92f4d562a0b7c5afa0fd0acca58625d1399c8612fca6"}
Oct 29 22:05:55.576456 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:55.576082    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:05:56.592006 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:56.586565    4550 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wrfmq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:05:56.592006 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:56.586629    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq" podUID="64621438-8bc0-40ce-8fe5-1129114a5fbf" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:05:56.843210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:56.841544    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:57.540165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:57.539954    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:57.692458 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:57.692426    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:57.846903 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:57.846853    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:58.183795 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:58.183721    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:59.526425 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:59.526394    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"
Oct 29 22:05:59.566644 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:59.566607    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:05:59.573496 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:59.573440    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq" podStartSLOduration=5.573424318 podStartE2EDuration="5.573424318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:55.664350378 +0000 UTC m=+1171.821475876" watchObservedRunningTime="2026-10-29 22:05:59.573424318 +0000 UTC m=+1175.730549816"
Oct 29 22:05:59.588620 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:05:59.588583    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrfmq"
Oct 29 22:06:00.940849 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:00.940809    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:01.510982 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:01.510946    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:01.546113 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:01.546085    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"
Oct 29 22:06:03.058595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:03.058548    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:03.234598 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:03.234562    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:03.389717 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:03.389677    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:04.863051 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:04.863016    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:06.137064 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:06.137030    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:06.318812 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:06.318775    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:06.487492 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:06.487384    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:06.707905 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:06.707854    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:07.772286 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:07.772250    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:08.397753 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:08.397714    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:08.677506 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:08.677415    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:08.914482 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:08.914437    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:10.291732 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:10.291695    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:10.416094 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:10.416063    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:10.942772 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:10.942724    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:11.496197 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:06:11.496158    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:06:12.169004 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:12.168971    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:12.767699 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:12.767670    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:12.900846 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:12.900818    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:13.019697 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:13.019610    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:14.777123 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:14.777081    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:14.964173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:14.963249    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:15.297880 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:15.297828    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:15.447028 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:15.446993    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:15.647453 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:15.647415    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:17.640685 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:17.631703    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:19.254071 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:19.254037    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:19.402843 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:19.402807    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:20.083490 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:20.083441    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:21.974869 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:21.974825    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:23.605566 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:23.605534    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:25.294614 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:25.294581    4550 scope.go:122] "RemoveContainer" containerID="c147dd074d943c076fee84eb13776e53b31750043d7b6e8c87fcc5c44e9a0a53"
Oct 29 22:06:25.340327 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:25.340301    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:27.361313 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:27.361160    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:29.020397 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:29.020340    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:30.771838 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:30.771795    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:32.542941 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:32.542906    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:34.580228 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:34.580194    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:35.919052 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:35.919008    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:36.063045 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:36.059503    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:37.191699 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:37.191625    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:38.424851 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:38.424820    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:39.010379 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:39.010343    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:40.876693 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:40.876646    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:41.491712 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:06:41.491676    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:06:42.560357 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:42.560324    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:42.712589 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:42.712556    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:44.234532 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:44.234499    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:46.129293 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:46.129257    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:47.032487 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:47.032435    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:47.639183 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:47.639108    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:47.836196 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:47.828333    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:48.634056 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:48.634017    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:50.460677 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:50.460641    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:50.596680 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:50.596049    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:51.393269 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:51.373001    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:51.561430 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:51.561330    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:51.564189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:51.564158    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"]
Oct 29 22:06:51.772222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:51.759736    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:51.966294 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:51.966244    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:51.975173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:51.966875    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:51.982173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:51.974106    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.036222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:51.974166    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.036222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:51.990651    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.061169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:51.990706    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.061169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:51.990729    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.061169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:51.990748    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:06:52.061169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:51.990770    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:06:52.061169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:52.035245    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:52.126165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:52.110566    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:06:52.160510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:52.160471    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.488064 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:52.488020    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:52.735883 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:52.735833    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.839068 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:52.742808    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.866122 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:52.864069    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:52.980164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:52.969557    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.990226 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:52.989331    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:06:53.022097 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.022055    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:53.022971 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.022943    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:06:53.064168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.049078    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:53.087936 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.053742    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:06:53.088766 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.053772    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded"
Oct 29 22:06:53.089258 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.053803    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:53.093162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.053823    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:06:53.267162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.053844    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:06:53.289179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.053873    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:06:53.289179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.053892    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:06:53.289179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.088448    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:06:53.289179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.286911    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:06:53.289179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.286962    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:06:53.289179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.287002    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:06:53.289179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.287043    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:06:53.289179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.287081    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:06:53.289179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.287117    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:06:53.289179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.162421    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:06:53.289179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.163085    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:53.332369 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.332339    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:06:53.332709 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.332689    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:06:53.390297 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.390201    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:53.434628 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.434592    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:53.447783 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.447747    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:53.619743 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.619705    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:06:53.619979 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.619813    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:06:53.841564 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.839097    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:06:53.864674 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.860548    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:06:53.966162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.964288    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 29 22:06:53.990260 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:53.987785    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.030953 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.022536    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.102670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.102561    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.244195 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.234727    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.273180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.245485    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.273180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.246540    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.273180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.261472    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.278614 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.278581    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:54.293864 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.293346    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.446258 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.433703    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:06:54.446258 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.433766    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:06:54.446258 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.435895    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:06:54.446258 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.436311    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:06:54.503491 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.503345    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.658268 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.569034    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:54.658268 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.623365    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.001173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.990696    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:55.001173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:54.993588    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.032565 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:55.032509    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.242166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:55.241223    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.242484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:55.242455    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.281245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:55.278694    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:06:55.281245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:55.278804    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.281245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:55.279660    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.281245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:55.279949    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.303379 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:55.303331    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.310531 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:55.310493    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.431255 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:55.431211    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.513302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:55.440217    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.691364 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:55.691272    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.715042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:55.715000    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.744169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:55.744119    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:06:56.151169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.148533    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:06:56.209168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.069254    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.209168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.013381    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.257430 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.256730    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.284196 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.284131    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.310168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.286300    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:06:56.310168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.299321    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.310168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.307125    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.353176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.332668    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.353176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.341247    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.422223 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.421299    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:06:56.422223 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.421372    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:06:56.507423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.507378    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.560898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.560860    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:06:56.583043 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.522238    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.583043 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.569163    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs" podUID="db60d8d1-7c00-47f5-bff6-a70d7456a389" containerName="payload-processing" containerID="cri-o://6555c9aeb9263beca1a77426fcde9e5950f53a6914fb70eba01c05e606220035" gracePeriod=25
Oct 29 22:06:56.611363 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.611328    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:06:56.616211 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.611626    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h" podUID="be3e55fd-83b9-4c52-a2a9-28da7d04477e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.750132 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.748851    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.763561 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.750874    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:06:56.842184 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.841692    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:06:56.842184 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:56.841759    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.124225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.123789    4550 generic.go:356] "Generic (PLEG): container finished" podID="db60d8d1-7c00-47f5-bff6-a70d7456a389" containerID="6555c9aeb9263beca1a77426fcde9e5950f53a6914fb70eba01c05e606220035" exitCode=0
Oct 29 22:06:57.124225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.124059    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs" event={"ID":"db60d8d1-7c00-47f5-bff6-a70d7456a389","Type":"ContainerDied","Data":"6555c9aeb9263beca1a77426fcde9e5950f53a6914fb70eba01c05e606220035"}
Oct 29 22:06:57.198202 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.197590    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.250417 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.250238    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.266297 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.266239    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.288277 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.288228    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:06:57.288902 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.288527    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.310365 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.310318    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:06:57.311085 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.311043    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:06:57.312618 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.312591    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.349229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.348643    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.572230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.558576    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:06:57.588167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.575112    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.665566 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.665528    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:06:57.665889 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.665842    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp" podUID="7fe0d723-a137-4c93-9657-ca9258b08a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.759341 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.759289    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:06:57.759898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.759401    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.839243 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:57.839202    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 29 22:06:58.123352 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.123247    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.214265 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.213293    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.254336 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.254292    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.273207 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.273158    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.325169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.299743    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.325169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.309671    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.370540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.327196    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.370540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.327569    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:06:58.370540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.327826    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:06:58.370540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.328112    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:06:58.408195 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.371004    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:06:58.595319 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.595272    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.614163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.597850    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:06:58.614666 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.611901    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.716220 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.711083    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded"
Oct 29 22:06:58.770319 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.770270    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.774165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:58.770876    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:06:59.215494 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.215454    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.266584 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.265231    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:06:59.316165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.295375    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:06:59.322344 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.322304    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:06:59.350638 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.350584    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.407700 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.352589    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:06:59.407700 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.352837    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.407700 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.374730    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.439192 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.409333    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:06:59.613515 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.607812    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.616010 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.614768    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.625758 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.625728    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"
Oct 29 22:06:59.655724 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.655688    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:06:59.655996 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.655967    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.735743 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.735705    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db60d8d1-7c00-47f5-bff6-a70d7456a389-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db60d8d1-7c00-47f5-bff6-a70d7456a389-plugins-config-volume\") pod \"db60d8d1-7c00-47f5-bff6-a70d7456a389\" (UID: \"db60d8d1-7c00-47f5-bff6-a70d7456a389\") "
Oct 29 22:06:59.753167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.748293    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db60d8d1-7c00-47f5-bff6-a70d7456a389-kube-api-access-9j4lp\" (UniqueName: \"kubernetes.io/projected/db60d8d1-7c00-47f5-bff6-a70d7456a389-kube-api-access-9j4lp\") pod \"db60d8d1-7c00-47f5-bff6-a70d7456a389\" (UID: \"db60d8d1-7c00-47f5-bff6-a70d7456a389\") "
Oct 29 22:06:59.782165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.747383    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db60d8d1-7c00-47f5-bff6-a70d7456a389-plugins-config-volume" pod "db60d8d1-7c00-47f5-bff6-a70d7456a389" (UID: "db60d8d1-7c00-47f5-bff6-a70d7456a389"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:06:59.782608 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.782543    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db60d8d1-7c00-47f5-bff6-a70d7456a389-kube-api-access-9j4lp" pod "db60d8d1-7c00-47f5-bff6-a70d7456a389" (UID: "db60d8d1-7c00-47f5-bff6-a70d7456a389"). InnerVolumeSpecName "kube-api-access-9j4lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:06:59.782741 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.772324    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.784755 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:06:59.784222    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.109931 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.109902    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j4lp\" (UniqueName: \"kubernetes.io/projected/db60d8d1-7c00-47f5-bff6-a70d7456a389-kube-api-access-9j4lp\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:07:00.217482 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.217452    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db60d8d1-7c00-47f5-bff6-a70d7456a389-plugins-config-volume\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:07:00.218972 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.218936    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.270792 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.270747    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.317291 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.316492    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.347993 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.332728    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.366391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.366281    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.370367 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.366749    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:07:00.370367 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.366804    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:07:00.380238 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.379414    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.446097 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.445507    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:07:00.532170 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.518284    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.532695 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.532650    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 29 22:07:00.619255 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.617238    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:07:00.619255 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.618624    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:07:00.699440 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.699395    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs" event={"ID":"db60d8d1-7c00-47f5-bff6-a70d7456a389","Type":"ContainerDied","Data":"d2fc62392a35321fd7e8296e49a777aeb5d78b60d2b22b2e0e69201bedccaa28"}
Oct 29 22:07:00.699805 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.699780    4550 scope.go:122] "RemoveContainer" containerID="6555c9aeb9263beca1a77426fcde9e5950f53a6914fb70eba01c05e606220035"
Oct 29 22:07:00.700209 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.700187    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"
Oct 29 22:07:00.735351 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.734996    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded"
Oct 29 22:07:00.787168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.785334    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:07:00.866171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.817278    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:07:00.874284 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.874201    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"]
Oct 29 22:07:00.945613 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:00.945571    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-7847cd8f7b-5f5qs"]
Oct 29 22:07:01.001289 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.001241    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" podUID="93b4187a-1551-4cf8-b807-d057baf03920" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.009436 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.009407    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:07:01.246355 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.246239    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.282730 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.282685    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.337308 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.337254    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.368006 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.367967    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:07:01.378263 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.378224    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.382495 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.378466    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.382495 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.380431    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.446288 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.446238    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.606522 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.606253    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.620943 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.620904    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:07:01.622579 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.622533    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:07:01.634929 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.634888    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:01.702711 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.702678    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:07:01.802227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.802185    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.867425 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.867300    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.963118 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.963069    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:01.963645 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:01.963600    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:02.242034 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.241944    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db60d8d1-7c00-47f5-bff6-a70d7456a389" path="/var/lib/kubelet/pods/db60d8d1-7c00-47f5-bff6-a70d7456a389/volumes"
Oct 29 22:07:02.330361 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.330314    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:02.331599 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.331560    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:07:02.350166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.348435    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:07:02.367318 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.367278    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:02.367629 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.367598    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:02.369794 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.369761    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:07:02.384349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.384309    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:07:02.398024 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.391214    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:02.398024 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.391610    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:07:02.446755 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.446708    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:02.639232 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.639065    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:02.640351 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.639471    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:02.707284 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.707233    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Oct 29 22:07:02.844167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.819111    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:07:02.869268 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:02.869224    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.003615 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.001315    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.054168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.053597    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.351166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.334899    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.351166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.335296    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:07:03.351166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.335586    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.351166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.336983    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:07:03.356186 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.355251    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.381167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.372038    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:07:03.393886 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.388378    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.393886 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.392081    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.393886 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.392433    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:07:03.602303 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.598575    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.602303 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.599125    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:03.602303 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.599186    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.649577 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.649522    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:07:03.677958 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.651767    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:07:03.798173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.797728    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:03.845171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.832400    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.845171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.844532    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.861156 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.860923    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:07:03.890220 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.873056    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:07:03.930177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:03.930041    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:07:04.054244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.053700    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.266230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.264977    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:07:04.340023 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.338881    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:07:04.340816 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.340782    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:07:04.367791 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.367749    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.378617 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.378579    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.423764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.422818    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:07:04.423966 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.423762    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:07:04.657348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.657300    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:07:04.658635 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.658603    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:07:04.687628 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.687590    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:04.687952 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.687902    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.711241 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.711066    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:04.711241 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.711129    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" podUID="a26797f1-f34c-44e3-86e9-dd0ec24af003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.851166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.849246    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.907181 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:04.896026    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.091726 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:05.082688    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.180675 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:05.177855    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.282793 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:05.282626    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.349218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:05.348976    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.349399 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:05.349361    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.368552 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:05.368513    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:07:05.394210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:05.394170    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:07:05.431373 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:05.431328    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.432232 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:05.432108    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.450346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:05.450304    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.677361 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:05.677257    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.854344 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:05.854296    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.910357 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:05.910301    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.918183 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:05.915824    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:07:05.944575 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:05.937606    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:07:06.177085 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.177053    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:07:06.255801 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.202668    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.296230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.296178    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" podUID="93b4187a-1551-4cf8-b807-d057baf03920" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.378456 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.378413    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.509008 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.508915    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:07:06.513393 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.513348    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.517850 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.366515    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.527685 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.407124    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.534604 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.531729    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.575161 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.573271    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:07:06.575161 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.573352    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:07:06.575161 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.573411    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:07:06.578192 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.578136    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:06.610174 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.609081    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:07:06.614343 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.610428    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:07:06.649987 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.645877    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:07:06.653618 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.653330    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.659166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.653853    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:07:06.659166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.658266    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:07:06.659166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.658301    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:07:06.659166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.658333    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:07:06.662587 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:06.661338    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:07:08.220313 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:08.220277    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:08.381026 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:08.380986    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:09.295289 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:09.295182    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:10.180074 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:10.180045    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:10.350838 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:10.350808    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:11.509305 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:07:11.509242    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:07:11.549825 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:11.549788    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:12.154839 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:12.150983    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:12.410296 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:12.410217    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:12.560495 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:12.560431    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:12.698593 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:12.698509    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:12.989552 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:12.989470    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:14.345604 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:14.345542    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:16.139436 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:16.139398    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:16.276966 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:16.276921    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:17.919525 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:17.919490    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"]
Oct 29 22:07:17.928301 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:17.928272    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:18.085874 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:18.085841    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:19.830506 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:19.830465    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:22.346000 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:22.345960    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:24.453878 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:24.453839    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:26.203369 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:26.203326    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:28.425893 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.425853    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:28.532707 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.532669    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"]
Oct 29 22:07:28.533420 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.533400    4550 cpu_manager.go:414] "containerMap: removing container" podUID="db60d8d1-7c00-47f5-bff6-a70d7456a389" containerName="payload-processing"
Oct 29 22:07:28.533420 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.533420    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db60d8d1-7c00-47f5-bff6-a70d7456a389" containerName="payload-processing"
Oct 29 22:07:28.533649 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.533635    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="db60d8d1-7c00-47f5-bff6-a70d7456a389" containerName="payload-processing"
Oct 29 22:07:28.538614 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.538596    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.579345 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.579307    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.579501 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.579377    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.579501 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.579404    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fbtf\" (UniqueName: \"kubernetes.io/projected/606c44a7-c952-49b1-b5eb-6f527d149c02-kube-api-access-5fbtf\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.579501 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.579429    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606c44a7-c952-49b1-b5eb-6f527d149c02-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.579704 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.579583    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.579704 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.579647    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.579810 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.579759    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.579884 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.579842    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.579884 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.579866    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.582988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.582964    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"]
Oct 29 22:07:28.630294 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.630264    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-380eea-openshift-default-dockercfg-lm8cf\""
Oct 29 22:07:28.681454 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.681373    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.681454 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.681421    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.681454 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.681441    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fbtf\" (UniqueName: \"kubernetes.io/projected/606c44a7-c952-49b1-b5eb-6f527d149c02-kube-api-access-5fbtf\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.681454 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.681457    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606c44a7-c952-49b1-b5eb-6f527d149c02-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.681844 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.681505    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.681844 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.681548    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.681844 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.681652    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.681844 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.681743    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.681844 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.681766    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.682111 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.682033    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.682310 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.682281    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.682600 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.682380    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606c44a7-c952-49b1-b5eb-6f527d149c02-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.682600 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.682526    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.682742 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.682710    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.684090 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.684070    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.684217 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.684075    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.713741 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.713715    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fbtf\" (UniqueName: \"kubernetes.io/projected/606c44a7-c952-49b1-b5eb-6f527d149c02-kube-api-access-5fbtf\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.728981 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.728956    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:28.854490 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:28.854457    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:29.195815 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:29.195788    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"]
Oct 29 22:07:29.196823 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:07:29.196796    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606c44a7_c952_49b1_b5eb_6f527d149c02.slice/crio-e865d432a1ae083e59afc1c373a040253b8261ff88bed8a906bb92430f1f1d8b WatchSource:0}: Error finding container e865d432a1ae083e59afc1c373a040253b8261ff88bed8a906bb92430f1f1d8b: Status 404 returned error can't find the container with id e865d432a1ae083e59afc1c373a040253b8261ff88bed8a906bb92430f1f1d8b
Oct 29 22:07:29.199360 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:29.199331    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:29.199432 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:29.199396    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:29.199432 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:29.199423    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:29.835821 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:29.835785    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz" event={"ID":"606c44a7-c952-49b1-b5eb-6f527d149c02","Type":"ContainerStarted","Data":"6d8d8a9ccd5ab2288670821dd9e99c207efbcee77811449a1bca29335d15aa73"}
Oct 29 22:07:29.836245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:29.835839    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz" event={"ID":"606c44a7-c952-49b1-b5eb-6f527d149c02","Type":"ContainerStarted","Data":"e865d432a1ae083e59afc1c373a040253b8261ff88bed8a906bb92430f1f1d8b"}
Oct 29 22:07:29.855678 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:29.855646    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:29.896446 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:29.896384    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz" podStartSLOduration=1.896365292 podStartE2EDuration="1.896365292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:29.895741467 +0000 UTC m=+1266.052866985" watchObservedRunningTime="2026-10-29 22:07:29.896365292 +0000 UTC m=+1266.053490793"
Oct 29 22:07:30.857038 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:30.857005    4550 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:07:30.857459 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:30.857064    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz" podUID="606c44a7-c952-49b1-b5eb-6f527d149c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 29 22:07:31.005163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:31.004694    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:31.855556 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:31.855524    4550 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:07:31.855736 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:31.855574    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz" podUID="606c44a7-c952-49b1-b5eb-6f527d149c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 29 22:07:32.594799 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:32.594769    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:32.881283 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:32.844386    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:32.881283 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:32.854985    4550 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:07:32.881283 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:32.855052    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz" podUID="606c44a7-c952-49b1-b5eb-6f527d149c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 29 22:07:32.941955 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:32.941893    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" containerID="cri-o://5723829c38825fe65209d59004490fcef8a0e4e1d350287467f56647da3818b0" gracePeriod=45
Oct 29 22:07:33.662169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.655085    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:07:33.754165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.749696    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-home\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-home\") pod \"89f0b9a2-8e74-4f05-9754-5f3b18440530\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") "
Oct 29 22:07:33.754165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.749786    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89f0b9a2-8e74-4f05-9754-5f3b18440530-tls-certs\" (UniqueName: \"kubernetes.io/secret/89f0b9a2-8e74-4f05-9754-5f3b18440530-tls-certs\") pod \"89f0b9a2-8e74-4f05-9754-5f3b18440530\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") "
Oct 29 22:07:33.754165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.749827    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-tmp-dir\") pod \"89f0b9a2-8e74-4f05-9754-5f3b18440530\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") "
Oct 29 22:07:33.754165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.749942    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-dshm\") pod \"89f0b9a2-8e74-4f05-9754-5f3b18440530\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") "
Oct 29 22:07:33.754165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.750053    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f0b9a2-8e74-4f05-9754-5f3b18440530-kube-api-access-fbb6c\" (UniqueName: \"kubernetes.io/projected/89f0b9a2-8e74-4f05-9754-5f3b18440530-kube-api-access-fbb6c\") pod \"89f0b9a2-8e74-4f05-9754-5f3b18440530\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") "
Oct 29 22:07:33.754165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.750544    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-model-cache\") pod \"89f0b9a2-8e74-4f05-9754-5f3b18440530\" (UID: \"89f0b9a2-8e74-4f05-9754-5f3b18440530\") "
Oct 29 22:07:33.754165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.751423    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-home" pod "89f0b9a2-8e74-4f05-9754-5f3b18440530" (UID: "89f0b9a2-8e74-4f05-9754-5f3b18440530"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:07:33.754165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.751594    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-tmp-dir" pod "89f0b9a2-8e74-4f05-9754-5f3b18440530" (UID: "89f0b9a2-8e74-4f05-9754-5f3b18440530"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:07:33.754165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.751783    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-model-cache" pod "89f0b9a2-8e74-4f05-9754-5f3b18440530" (UID: "89f0b9a2-8e74-4f05-9754-5f3b18440530"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:07:33.792418 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.783989    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f0b9a2-8e74-4f05-9754-5f3b18440530-kube-api-access-fbb6c" pod "89f0b9a2-8e74-4f05-9754-5f3b18440530" (UID: "89f0b9a2-8e74-4f05-9754-5f3b18440530"). InnerVolumeSpecName "kube-api-access-fbb6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:07:33.825214 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.806430    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f0b9a2-8e74-4f05-9754-5f3b18440530-tls-certs" pod "89f0b9a2-8e74-4f05-9754-5f3b18440530" (UID: "89f0b9a2-8e74-4f05-9754-5f3b18440530"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:07:33.839461 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.839354    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-dshm" pod "89f0b9a2-8e74-4f05-9754-5f3b18440530" (UID: "89f0b9a2-8e74-4f05-9754-5f3b18440530"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:07:33.854089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.853911    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-home\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:07:33.854089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.853945    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f0b9a2-8e74-4f05-9754-5f3b18440530-tls-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:07:33.854089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.853960    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-tmp-dir\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:07:33.854089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.853972    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-dshm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:07:33.854089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.853985    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbb6c\" (UniqueName: \"kubernetes.io/projected/89f0b9a2-8e74-4f05-9754-5f3b18440530-kube-api-access-fbb6c\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:07:33.854089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.854000    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f0b9a2-8e74-4f05-9754-5f3b18440530-model-cache\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:07:33.855053 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.855011    4550 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:07:33.855284 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.855257    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz" podUID="606c44a7-c952-49b1-b5eb-6f527d149c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 29 22:07:33.931626 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.931590    4550 generic.go:356] "Generic (PLEG): container finished" podID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerID="5723829c38825fe65209d59004490fcef8a0e4e1d350287467f56647da3818b0" exitCode=0
Oct 29 22:07:33.932290 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.931738    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" event={"ID":"89f0b9a2-8e74-4f05-9754-5f3b18440530","Type":"ContainerDied","Data":"5723829c38825fe65209d59004490fcef8a0e4e1d350287467f56647da3818b0"}
Oct 29 22:07:33.932290 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.931765    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" event={"ID":"89f0b9a2-8e74-4f05-9754-5f3b18440530","Type":"ContainerDied","Data":"2fa28136c8de0829e805d6ff7267e66bfdc8295c46b1d05e93988ac45278d326"}
Oct 29 22:07:33.932290 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.931780    4550 scope.go:122] "RemoveContainer" containerID="5723829c38825fe65209d59004490fcef8a0e4e1d350287467f56647da3818b0"
Oct 29 22:07:33.932290 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.931893    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"
Oct 29 22:07:33.945547 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.945390    4550 scope.go:122] "RemoveContainer" containerID="5723829c38825fe65209d59004490fcef8a0e4e1d350287467f56647da3818b0"
Oct 29 22:07:33.945710 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:07:33.945690    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5723829c38825fe65209d59004490fcef8a0e4e1d350287467f56647da3818b0\": container with ID starting with 5723829c38825fe65209d59004490fcef8a0e4e1d350287467f56647da3818b0 not found: ID does not exist" containerID="5723829c38825fe65209d59004490fcef8a0e4e1d350287467f56647da3818b0"
Oct 29 22:07:33.945771 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:33.945722    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5723829c38825fe65209d59004490fcef8a0e4e1d350287467f56647da3818b0"} err="failed to get container status \"5723829c38825fe65209d59004490fcef8a0e4e1d350287467f56647da3818b0\": rpc error: code = NotFound desc = could not find container \"5723829c38825fe65209d59004490fcef8a0e4e1d350287467f56647da3818b0\": container with ID starting with 5723829c38825fe65209d59004490fcef8a0e4e1d350287467f56647da3818b0 not found: ID does not exist"
Oct 29 22:07:34.008734 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:34.008699    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"]
Oct 29 22:07:34.032114 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:34.032070    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv"]
Oct 29 22:07:34.140776 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:34.140675    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:34.344713 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:34.344667    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-751a7416/e2e-lifecycle-model-751a7416-kserve-5975f56877-xdhxv" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:34.855850 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:34.855812    4550 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:07:34.856334 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:34.855866    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz" podUID="606c44a7-c952-49b1-b5eb-6f527d149c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 29 22:07:35.027766 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:35.023991    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:35.237374 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:35.237282    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:35.856900 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:35.856861    4550 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:07:35.857819 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:35.857777    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz" podUID="606c44a7-c952-49b1-b5eb-6f527d149c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 29 22:07:36.051041 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:36.051003    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" path="/var/lib/kubelet/pods/89f0b9a2-8e74-4f05-9754-5f3b18440530/volumes"
Oct 29 22:07:36.591418 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:36.591376    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:36.859843 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:36.859763    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:36.860297 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:36.860165    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:36.860997 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:36.860977    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:07:39.398719 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.398678    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:39.730452 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.730370    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"]
Oct 29 22:07:39.731082 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.731068    4550 cpu_manager.go:414] "containerMap: removing container" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main"
Oct 29 22:07:39.731127 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.731083    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main"
Oct 29 22:07:39.731203 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.731196    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f0b9a2-8e74-4f05-9754-5f3b18440530" containerName="main"
Oct 29 22:07:39.735905 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.735877    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.755006 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.754978    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-dd38c2-openshift-default-dockercfg-bqcds\""
Oct 29 22:07:39.770256 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.770231    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"]
Oct 29 22:07:39.790193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.790155    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.790367 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.790235    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.790367 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.790346    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.790653 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.790627    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjcj2\" (UniqueName: \"kubernetes.io/projected/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-kube-api-access-tjcj2\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.790808 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.790786    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.790870 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.790828    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.790922 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.790867    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.790978 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.790925    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.791061 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.791042    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.892186 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.892130    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.892350 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.892203    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.892350 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.892234    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.892350 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.892271    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.892462 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.892374    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.892462 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.892425    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.892564 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.892491    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.892613 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.892557    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.892613 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.892578    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.892717 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.892646    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.892717 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.892673    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.892829 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.892793    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjcj2\" (UniqueName: \"kubernetes.io/projected/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-kube-api-access-tjcj2\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.892932 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.892883    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.893370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.893343    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.894604 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.894579    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.894922 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.894902    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.921136 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.921109    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjcj2\" (UniqueName: \"kubernetes.io/projected/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-kube-api-access-tjcj2\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:39.921602 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:39.921416    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:40.053923 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:40.053852    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:40.413952 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:40.413903    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"]
Oct 29 22:07:40.415322 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:07:40.415293    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3bd4c5a_b9d8_4c9d_8dee_4114edb6ffa7.slice/crio-d0eb07e481b259e8ce8aa82673c3de577305c22adb4d26e900546bd25def208b WatchSource:0}: Error finding container d0eb07e481b259e8ce8aa82673c3de577305c22adb4d26e900546bd25def208b: Status 404 returned error can't find the container with id d0eb07e481b259e8ce8aa82673c3de577305c22adb4d26e900546bd25def208b
Oct 29 22:07:40.418038 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:40.418006    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:40.418122 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:40.418072    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:40.418122 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:40.418103    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:41.031271 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.031231    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg" event={"ID":"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7","Type":"ContainerStarted","Data":"6f3ee1dcbbb8e825cdc92cfb70a9856c368824bf45f96399e1181b71728a6058"}
Oct 29 22:07:41.031487 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.031298    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg" event={"ID":"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7","Type":"ContainerStarted","Data":"d0eb07e481b259e8ce8aa82673c3de577305c22adb4d26e900546bd25def208b"}
Oct 29 22:07:41.054249 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.054212    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:41.056099 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.056072    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:07:41.056274 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.056120    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg" podUID="b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:07:41.452590 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.452539    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg" podStartSLOduration=2.452516192 podStartE2EDuration="2.452516192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:41.109945162 +0000 UTC m=+1277.267070661" watchObservedRunningTime="2026-10-29 22:07:41.452516192 +0000 UTC m=+1277.609641689"
Oct 29 22:07:41.455750 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.455724    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"]
Oct 29 22:07:41.460651 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.460230    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:41.491194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.485540    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-edc2d3-serving-cert\""
Oct 29 22:07:41.509284 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.509254    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"]
Oct 29 22:07:41.526196 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.526134    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjlc\" (UniqueName: \"kubernetes.io/projected/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-kube-api-access-pvjlc\") pod \"maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:41.526389 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.526375    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-metrics-certs\") pod \"maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:41.526458 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.526445    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:41.526501 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.526490    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:41.542349 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:07:41.540585    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:07:41.559044 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.558968    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:41.627885 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.627615    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-metrics-certs\") pod \"maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:41.630036 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.630010    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:41.630293 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.630273    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:41.630547 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.630527    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjlc\" (UniqueName: \"kubernetes.io/projected/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-kube-api-access-pvjlc\") pod \"maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:41.631099 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:07:41.631082    4550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-serving-cert: secret "maas-api-e2e-ipp-a-edc2d3-serving-cert" not found
Oct 29 22:07:41.631309 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:07:41.631298    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-maas-api-tls podName:3b9c227f-4871-457c-a0fd-5893b6bcd0ba nodeName:}" failed. No retries permitted until 2026-10-29 22:07:42.131277843 +0000 UTC m=+1278.288403324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-maas-api-tls") pod "maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs" (UID: "3b9c227f-4871-457c-a0fd-5893b6bcd0ba") : secret "maas-api-e2e-ipp-a-edc2d3-serving-cert" not found
Oct 29 22:07:41.632030 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.632012    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:41.635081 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.635030    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-metrics-certs\") pod \"maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:41.725225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.721652    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:41.751717 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.751657    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjlc\" (UniqueName: \"kubernetes.io/projected/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-kube-api-access-pvjlc\") pod \"maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:41.898230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:41.890817    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:42.055287 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:42.055209    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:07:42.055433 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:42.055294    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg" podUID="b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:07:42.158464 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:42.158428    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:42.165165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:42.161552    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:42.411370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:42.411330    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:42.807987 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:07:42.807908    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9c227f_4871_457c_a0fd_5893b6bcd0ba.slice/crio-72025eaaaf41d0ffcccd57356fecadd5e0bc969168a7b81d59f36627104116f6 WatchSource:0}: Error finding container 72025eaaaf41d0ffcccd57356fecadd5e0bc969168a7b81d59f36627104116f6: Status 404 returned error can't find the container with id 72025eaaaf41d0ffcccd57356fecadd5e0bc969168a7b81d59f36627104116f6
Oct 29 22:07:42.812966 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:42.812883    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"]
Oct 29 22:07:43.055644 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.055551    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:07:43.055990 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.055935    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg" podUID="b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:07:43.149370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.148312    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs" event={"ID":"3b9c227f-4871-457c-a0fd-5893b6bcd0ba","Type":"ContainerStarted","Data":"72025eaaaf41d0ffcccd57356fecadd5e0bc969168a7b81d59f36627104116f6"}
Oct 29 22:07:43.595946 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.595910    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"]
Oct 29 22:07:43.601122 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.601100    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.611247 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.611185    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"]
Oct 29 22:07:43.735164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.734257    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9q95\" (UniqueName: \"kubernetes.io/projected/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-kube-api-access-f9q95\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.735164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.734324    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.735164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.734366    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.735164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.734466    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.735164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.734536    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.735164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.734575    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.735164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.734705    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.735164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.734749    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.735164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.734792    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.836534 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.836491    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9q95\" (UniqueName: \"kubernetes.io/projected/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-kube-api-access-f9q95\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.837042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.836555    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.837042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.836590    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.837042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.836687    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.837042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.836753    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.837042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.836804    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.837042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.836942    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.837042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.836986    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.837042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.837026    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.837425 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.837122    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.837425 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.837175    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.837425 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.837021    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.837425 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.837225    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.837741 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.837722    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.839124 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.839095    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.839939 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.839912    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.860754 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.860694    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.860917 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.860893    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9q95\" (UniqueName: \"kubernetes.io/projected/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-kube-api-access-f9q95\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.940337 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.940306    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:43.954411 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:43.954288    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:44.054318 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:44.054279    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:07:44.054474 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:44.054356    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg" podUID="b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:07:44.203032 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:44.201232    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs" event={"ID":"3b9c227f-4871-457c-a0fd-5893b6bcd0ba","Type":"ContainerStarted","Data":"12d58eb5b42e2ad632726b8ac42c80e70bc57ba41e023a79ee05b03768777628"}
Oct 29 22:07:44.203311 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:44.203280    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:44.252009 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:44.251950    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs" podStartSLOduration=2.803488849 podStartE2EDuration="3.251933778s" totalImagesPullingTime="448.444929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:44.249252954 +0000 UTC m=+1280.406378459" watchObservedRunningTime="2026-10-29 22:07:44.251933778 +0000 UTC m=+1280.409059278"
Oct 29 22:07:44.398183 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:44.398123    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"]
Oct 29 22:07:44.414739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:44.414694    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:44.414891 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:44.414773    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:44.414891 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:44.414801    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:44.503358 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:44.502929    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:44.503358 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:44.502992    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w" podUID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:45.054614 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:45.054577    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:07:45.055106 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:45.054629    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg" podUID="b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:07:45.236743 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:45.236702    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55" event={"ID":"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757","Type":"ContainerStarted","Data":"b862f9c98dbe483f54d78b0d306fff152e55854b8402cc84177057b361f195b7"}
Oct 29 22:07:45.236952 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:45.236771    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55" event={"ID":"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757","Type":"ContainerStarted","Data":"8db4358e39fd03022e7f2acb83658d69968a2f7dbfe305b0062ca24c1ec18e0c"}
Oct 29 22:07:45.302407 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:45.302338    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55" podStartSLOduration=2.302318837 podStartE2EDuration="2.302318837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:45.2939334 +0000 UTC m=+1281.451058899" watchObservedRunningTime="2026-10-29 22:07:45.302318837 +0000 UTC m=+1281.459444339"
Oct 29 22:07:45.943769 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:45.940394    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:45.943769 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:45.942000    4550 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:07:45.943769 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:45.942046    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55" podUID="06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:07:46.054833 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:46.054801    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:07:46.055245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:46.054848    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg" podUID="b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:07:46.227190 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:46.225048    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:46.399759 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:46.397190    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:46.419167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:46.415365    4550 generic.go:356] "Generic (PLEG): container finished" podID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerID="a2746305a38ab0891c511dd8ef7e79ff52f59c93741d58dc873b85ca742988bb" exitCode=0
Oct 29 22:07:46.419167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:46.417165    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w" event={"ID":"097cc026-11a8-45b2-8eda-26fce58e7a02","Type":"ContainerDied","Data":"a2746305a38ab0891c511dd8ef7e79ff52f59c93741d58dc873b85ca742988bb"}
Oct 29 22:07:46.436478 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:46.429844    4550 scope.go:122] "RemoveContainer" containerID="a2746305a38ab0891c511dd8ef7e79ff52f59c93741d58dc873b85ca742988bb"
Oct 29 22:07:46.538187 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:46.531946    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:46.538187 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:46.532057    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:46.538187 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:46.532111    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:46.647589 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:46.647545    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:46.940935 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:46.940895    4550 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:07:46.941104 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:46.940964    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55" podUID="06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:07:47.060905 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:47.060825    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:47.061458 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:47.061435    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:47.081164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:47.074410    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:07:47.429830 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:47.429788    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w" event={"ID":"097cc026-11a8-45b2-8eda-26fce58e7a02","Type":"ContainerStarted","Data":"96088a4b69e2f13ec068d23436a627ea1ae04a48f43d8ee362fe4cf742f200bf"}
Oct 29 22:07:47.951205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:47.946559    4550 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:07:47.951205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:47.946604    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55" podUID="06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:07:48.259343 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:48.257069    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:07:48.259343 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:48.257377    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:07:48.259343 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:48.257468    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w" podUID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:07:48.642450 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:48.642412    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:48.941959 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:48.941874    4550 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:07:48.941959 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:48.941927    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55" podUID="06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:07:49.251081 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:49.250984    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:07:49.251388 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:49.251346    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w" podUID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:07:49.940671 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:49.940630    4550 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:07:49.941059 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:49.940695    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55" podUID="06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:07:50.240788 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:50.240718    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:07:50.249939 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:50.249907    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:07:50.250077 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:50.249970    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w" podUID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:07:50.406163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:50.406112    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:50.949571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:50.949534    4550 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:07:50.950217 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:50.950183    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55" podUID="06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:07:51.170767 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:51.170728    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:51.258974 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:51.258882    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:07:51.259281 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:51.259247    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w" podUID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:07:51.592176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:51.591656    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:51.794346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:51.794306    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:51.956302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:51.956225    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:51.956857 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:51.956719    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:51.957644 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:51.957624    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:07:52.135037 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:52.135004    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:52.250216 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:52.250120    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:07:52.250216 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:52.250193    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w" podUID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:07:53.254231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:53.251749    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:07:53.254231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:53.251953    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w" podUID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:07:53.816681 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:53.816631    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:54.255352 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:54.255322    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:07:54.256845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:54.256823    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:07:54.258033 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:54.258013    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:07:54.872933 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:54.872898    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"]
Oct 29 22:07:54.876542 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:54.876509    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:54.897426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:54.897395    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"]
Oct 29 22:07:54.907590 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:54.905989    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:54.910423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:54.910399    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"]
Oct 29 22:07:54.932307 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:54.931315    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"]
Oct 29 22:07:55.005012 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.004967    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:55.032660 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.032612    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnvft\" (UniqueName: \"kubernetes.io/projected/7c72b9b9-b934-4279-960c-d69fadadb664-kube-api-access-wnvft\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.032865 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.032676    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.032865 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.032770    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07ed650-5c07-4440-b64d-cb65a9529625-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.032865 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.032834    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.032983 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.032874    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.032983 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.032903    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.032983 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.032954    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.033076 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.033011    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07ed650-5c07-4440-b64d-cb65a9529625-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.033115 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.033085    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.033173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.033128    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.033247 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.033174    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.033247 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.033231    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsrk8\" (UniqueName: \"kubernetes.io/projected/a07ed650-5c07-4440-b64d-cb65a9529625-kube-api-access-xsrk8\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.033545 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.033514    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07ed650-5c07-4440-b64d-cb65a9529625-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.033678 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.033585    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.033678 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.033646    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.033793 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.033693    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c72b9b9-b934-4279-960c-d69fadadb664-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.033793 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.033752    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c72b9b9-b934-4279-960c-d69fadadb664-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.033894 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.033804    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c72b9b9-b934-4279-960c-d69fadadb664-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.135955 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.135836    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07ed650-5c07-4440-b64d-cb65a9529625-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.136119 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.135982    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.136119 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.136071    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.136273 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.136167    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c72b9b9-b934-4279-960c-d69fadadb664-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.136273 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.136257    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c72b9b9-b934-4279-960c-d69fadadb664-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.136351 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.136304    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c72b9b9-b934-4279-960c-d69fadadb664-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.136389 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.136350    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnvft\" (UniqueName: \"kubernetes.io/projected/7c72b9b9-b934-4279-960c-d69fadadb664-kube-api-access-wnvft\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.136467 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.136443    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.136681 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.136656    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07ed650-5c07-4440-b64d-cb65a9529625-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.136756 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.136741    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.136802 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.136790    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.136838 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.136823    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.136898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.136883    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.136963 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.136949    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07ed650-5c07-4440-b64d-cb65a9529625-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.137028 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.137014    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.137078 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.137065    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.137121 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.137101    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.137198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.137182    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsrk8\" (UniqueName: \"kubernetes.io/projected/a07ed650-5c07-4440-b64d-cb65a9529625-kube-api-access-xsrk8\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.137286 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.137255    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07ed650-5c07-4440-b64d-cb65a9529625-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.140763 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.137954    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.140763 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.138470    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.140763 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.140071    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.141110 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.141036    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.141660 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.141628    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c72b9b9-b934-4279-960c-d69fadadb664-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.144242 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.144198    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.145920 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.145329    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.147568 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.147251    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.147568 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.147284    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.148198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.148175    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.148446 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.148413    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07ed650-5c07-4440-b64d-cb65a9529625-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.148939 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.148909    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c72b9b9-b934-4279-960c-d69fadadb664-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.153175 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.153111    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.161755 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.161719    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07ed650-5c07-4440-b64d-cb65a9529625-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.165066 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.165026    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c72b9b9-b934-4279-960c-d69fadadb664-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.165842 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.165811    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsrk8\" (UniqueName: \"kubernetes.io/projected/a07ed650-5c07-4440-b64d-cb65a9529625-kube-api-access-xsrk8\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.166198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.166170    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnvft\" (UniqueName: \"kubernetes.io/projected/7c72b9b9-b934-4279-960c-d69fadadb664-kube-api-access-wnvft\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.197300 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.197259    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:55.241656 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.241340    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:55.563684 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:07:55.563651    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07ed650_5c07_4440_b64d_cb65a9529625.slice/crio-feeb7946778e2df4211ff6ab51e9abacf0e53ab204c8f34641afeeaf02c9c5e1 WatchSource:0}: Error finding container feeb7946778e2df4211ff6ab51e9abacf0e53ab204c8f34641afeeaf02c9c5e1: Status 404 returned error can't find the container with id feeb7946778e2df4211ff6ab51e9abacf0e53ab204c8f34641afeeaf02c9c5e1
Oct 29 22:07:55.564061 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.563996    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"]
Oct 29 22:07:55.566482 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.566448    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:55.566596 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.566508    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:55.566596 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.566537    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:55.639229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.639204    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"]
Oct 29 22:07:55.639515 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:07:55.639488    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c72b9b9_b934_4279_960c_d69fadadb664.slice/crio-243b7938b8491486f92bcf850f6544f92ab69bfac9df96a87c7941a8714bbabf WatchSource:0}: Error finding container 243b7938b8491486f92bcf850f6544f92ab69bfac9df96a87c7941a8714bbabf: Status 404 returned error can't find the container with id 243b7938b8491486f92bcf850f6544f92ab69bfac9df96a87c7941a8714bbabf
Oct 29 22:07:55.642622 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.642588    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:55.642716 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.642664    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:55.642716 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.642706    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:07:55.714125 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:55.714049    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:56.545927 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:56.545891    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8" event={"ID":"7c72b9b9-b934-4279-960c-d69fadadb664","Type":"ContainerStarted","Data":"28f101b46402fa9ee6400a5e40a8254b1cacfe618b54879cee5f349aa9c06a57"}
Oct 29 22:07:56.546132 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:56.545949    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8" event={"ID":"7c72b9b9-b934-4279-960c-d69fadadb664","Type":"ContainerStarted","Data":"243b7938b8491486f92bcf850f6544f92ab69bfac9df96a87c7941a8714bbabf"}
Oct 29 22:07:56.548054 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:56.548021    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq" event={"ID":"a07ed650-5c07-4440-b64d-cb65a9529625","Type":"ContainerStarted","Data":"9b537ee11bf475283e067cdbaa66a2cac75bf40ec171a77e6ef3e1d4d8b0b437"}
Oct 29 22:07:56.548202 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:56.548069    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq" event={"ID":"a07ed650-5c07-4440-b64d-cb65a9529625","Type":"ContainerStarted","Data":"feeb7946778e2df4211ff6ab51e9abacf0e53ab204c8f34641afeeaf02c9c5e1"}
Oct 29 22:07:56.619101 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:56.619047    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8" podStartSLOduration=2.619031459 podStartE2EDuration="2.619031459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:56.606729452 +0000 UTC m=+1292.763854950" watchObservedRunningTime="2026-10-29 22:07:56.619031459 +0000 UTC m=+1292.776156957"
Oct 29 22:07:56.655787 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:56.655709    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq" podStartSLOduration=2.65569348 podStartE2EDuration="2.65569348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:56.652641736 +0000 UTC m=+1292.809767228" watchObservedRunningTime="2026-10-29 22:07:56.65569348 +0000 UTC m=+1292.812818978"
Oct 29 22:07:57.199169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:57.198548    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:07:57.213768 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:57.213430    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:07:57.213768 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:57.213486    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq" podUID="a07ed650-5c07-4440-b64d-cb65a9529625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:07:57.242600 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:57.242566    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:07:57.244796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:57.244169    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:07:57.244796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:57.244211    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8" podUID="7c72b9b9-b934-4279-960c-d69fadadb664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:07:57.906072 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:57.906036    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:58.160399 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:58.160272    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:58.200307 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:58.200266    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:07:58.200643 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:58.200616    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq" podUID="a07ed650-5c07-4440-b64d-cb65a9529625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:07:58.243127 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:58.242771    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:07:58.243127 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:58.242818    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8" podUID="7c72b9b9-b934-4279-960c-d69fadadb664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:07:58.617165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:58.617001    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:07:59.225572 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:59.225306    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:07:59.225572 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:59.225379    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq" podUID="a07ed650-5c07-4440-b64d-cb65a9529625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:07:59.243069 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:59.243035    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:07:59.243245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:07:59.243105    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8" podUID="7c72b9b9-b934-4279-960c-d69fadadb664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:08:00.201585 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:00.201542    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:08:00.201903 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:00.201872    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq" podUID="a07ed650-5c07-4440-b64d-cb65a9529625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:08:00.243216 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:00.243174    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:08:00.243702 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:00.243230    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8" podUID="7c72b9b9-b934-4279-960c-d69fadadb664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:08:00.328194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:00.326003    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:00.522231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:00.519396    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:00.806226 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:00.806069    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:01.202174 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.201794    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:08:01.202174 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.201859    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq" podUID="a07ed650-5c07-4440-b64d-cb65a9529625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:08:01.295176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.281795    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:08:01.295176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.281855    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8" podUID="7c72b9b9-b934-4279-960c-d69fadadb664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:08:01.533830 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.531385    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"]
Oct 29 22:08:01.544736 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.541109    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"
Oct 29 22:08:01.544736 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.544134    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"]
Oct 29 22:08:01.557019 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.556987    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:08:01.589544 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.573323    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:08:01.624453 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.624413    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c52c5564-6eef-4835-8f36-2f378dea76fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd\" (UID: \"c52c5564-6eef-4835-8f36-2f378dea76fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"
Oct 29 22:08:01.624908 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.624882    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c52c5564-6eef-4835-8f36-2f378dea76fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd\" (UID: \"c52c5564-6eef-4835-8f36-2f378dea76fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"
Oct 29 22:08:01.624986 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.624937    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvx2m\" (UniqueName: \"kubernetes.io/projected/c52c5564-6eef-4835-8f36-2f378dea76fb-kube-api-access-fvx2m\") pod \"maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd\" (UID: \"c52c5564-6eef-4835-8f36-2f378dea76fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"
Oct 29 22:08:01.722804 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.722775    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:01.726380 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.726354    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvx2m\" (UniqueName: \"kubernetes.io/projected/c52c5564-6eef-4835-8f36-2f378dea76fb-kube-api-access-fvx2m\") pod \"maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd\" (UID: \"c52c5564-6eef-4835-8f36-2f378dea76fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"
Oct 29 22:08:01.726517 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.726435    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c52c5564-6eef-4835-8f36-2f378dea76fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd\" (UID: \"c52c5564-6eef-4835-8f36-2f378dea76fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"
Oct 29 22:08:01.727172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.726700    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c52c5564-6eef-4835-8f36-2f378dea76fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd\" (UID: \"c52c5564-6eef-4835-8f36-2f378dea76fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"
Oct 29 22:08:01.727373 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.727352    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c52c5564-6eef-4835-8f36-2f378dea76fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd\" (UID: \"c52c5564-6eef-4835-8f36-2f378dea76fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"
Oct 29 22:08:01.730332 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.730303    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c52c5564-6eef-4835-8f36-2f378dea76fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd\" (UID: \"c52c5564-6eef-4835-8f36-2f378dea76fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"
Oct 29 22:08:01.750742 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.750717    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvx2m\" (UniqueName: \"kubernetes.io/projected/c52c5564-6eef-4835-8f36-2f378dea76fb-kube-api-access-fvx2m\") pod \"maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd\" (UID: \"c52c5564-6eef-4835-8f36-2f378dea76fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"
Oct 29 22:08:01.880848 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:01.879681    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"
Oct 29 22:08:02.214168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:02.211243    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:08:02.214168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:02.211305    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq" podUID="a07ed650-5c07-4440-b64d-cb65a9529625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:08:02.245439 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:02.245408    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:08:02.245705 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:02.245676    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8" podUID="7c72b9b9-b934-4279-960c-d69fadadb664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:08:02.532286 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:02.529489    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"]
Oct 29 22:08:02.532286 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:08:02.530831    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52c5564_6eef_4835_8f36_2f378dea76fb.slice/crio-2538364d779d4022fa4e4d2e567b02979d66aff2199ecb1f97b21288f2c6276e WatchSource:0}: Error finding container 2538364d779d4022fa4e4d2e567b02979d66aff2199ecb1f97b21288f2c6276e: Status 404 returned error can't find the container with id 2538364d779d4022fa4e4d2e567b02979d66aff2199ecb1f97b21288f2c6276e
Oct 29 22:08:03.155388 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:03.155293    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd" event={"ID":"c52c5564-6eef-4835-8f36-2f378dea76fb","Type":"ContainerStarted","Data":"e7f9be395d8e152ddb2fbbcedcaacd28d383fc7ab90c8331751d147d2d6b98b5"}
Oct 29 22:08:03.155388 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:03.155349    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd" event={"ID":"c52c5564-6eef-4835-8f36-2f378dea76fb","Type":"ContainerStarted","Data":"2538364d779d4022fa4e4d2e567b02979d66aff2199ecb1f97b21288f2c6276e"}
Oct 29 22:08:03.202676 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:03.202646    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:08:03.203153 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:03.203113    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:08:03.203996 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:03.203974    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:08:03.212423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:03.212375    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd" podStartSLOduration=2.212363952 podStartE2EDuration="2.212363952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:03.198075751 +0000 UTC m=+1299.355201264" watchObservedRunningTime="2026-10-29 22:08:03.212363952 +0000 UTC m=+1299.369489449"
Oct 29 22:08:03.248922 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:03.248886    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:08:03.250205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:03.250030    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:08:03.252459 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:03.252436    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:08:03.912807 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:03.912714    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:04.163868 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:04.163779    4550 generic.go:356] "Generic (PLEG): container finished" podID="c52c5564-6eef-4835-8f36-2f378dea76fb" containerID="e7f9be395d8e152ddb2fbbcedcaacd28d383fc7ab90c8331751d147d2d6b98b5" exitCode=0
Oct 29 22:08:04.164007 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:04.163860    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd" event={"ID":"c52c5564-6eef-4835-8f36-2f378dea76fb","Type":"ContainerDied","Data":"e7f9be395d8e152ddb2fbbcedcaacd28d383fc7ab90c8331751d147d2d6b98b5"}
Oct 29 22:08:05.356154 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:05.356119    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"
Oct 29 22:08:05.391781 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:05.391755    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52c5564-6eef-4835-8f36-2f378dea76fb-kube-api-access-fvx2m\" (UniqueName: \"kubernetes.io/projected/c52c5564-6eef-4835-8f36-2f378dea76fb-kube-api-access-fvx2m\") pod \"c52c5564-6eef-4835-8f36-2f378dea76fb\" (UID: \"c52c5564-6eef-4835-8f36-2f378dea76fb\") "
Oct 29 22:08:05.391927 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:05.391822    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52c5564-6eef-4835-8f36-2f378dea76fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c52c5564-6eef-4835-8f36-2f378dea76fb-maas-api-openshift-ca\") pod \"c52c5564-6eef-4835-8f36-2f378dea76fb\" (UID: \"c52c5564-6eef-4835-8f36-2f378dea76fb\") "
Oct 29 22:08:05.392055 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:05.392036    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c52c5564-6eef-4835-8f36-2f378dea76fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c52c5564-6eef-4835-8f36-2f378dea76fb-maas-api-xks-ca\") pod \"c52c5564-6eef-4835-8f36-2f378dea76fb\" (UID: \"c52c5564-6eef-4835-8f36-2f378dea76fb\") "
Oct 29 22:08:05.392288 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:05.392265    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52c5564-6eef-4835-8f36-2f378dea76fb-maas-api-openshift-ca" pod "c52c5564-6eef-4835-8f36-2f378dea76fb" (UID: "c52c5564-6eef-4835-8f36-2f378dea76fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:05.392845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:05.392823    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c52c5564-6eef-4835-8f36-2f378dea76fb-maas-api-openshift-ca\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:05.393902 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:05.393876    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52c5564-6eef-4835-8f36-2f378dea76fb-kube-api-access-fvx2m" pod "c52c5564-6eef-4835-8f36-2f378dea76fb" (UID: "c52c5564-6eef-4835-8f36-2f378dea76fb"). InnerVolumeSpecName "kube-api-access-fvx2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:05.394091 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:05.394063    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c52c5564-6eef-4835-8f36-2f378dea76fb-maas-api-xks-ca" pod "c52c5564-6eef-4835-8f36-2f378dea76fb" (UID: "c52c5564-6eef-4835-8f36-2f378dea76fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:08:05.495082 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:05.494992    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c52c5564-6eef-4835-8f36-2f378dea76fb-maas-api-xks-ca\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:05.495082 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:05.495032    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvx2m\" (UniqueName: \"kubernetes.io/projected/c52c5564-6eef-4835-8f36-2f378dea76fb-kube-api-access-fvx2m\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:05.740470 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:05.740439    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:05.866402 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:05.866351    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:05.991708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:05.991671    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:06.198574 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:06.198497    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"
Oct 29 22:08:06.198574 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:06.198525    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd" event={"ID":"c52c5564-6eef-4835-8f36-2f378dea76fb","Type":"ContainerDied","Data":"2538364d779d4022fa4e4d2e567b02979d66aff2199ecb1f97b21288f2c6276e"}
Oct 29 22:08:06.198796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:06.198585    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2538364d779d4022fa4e4d2e567b02979d66aff2199ecb1f97b21288f2c6276e"
Oct 29 22:08:06.209390 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:06.209352    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"]
Oct 29 22:08:06.209613 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:06.209586    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9" podUID="73ce2b92-3933-439a-b273-d313d80ecaf1" containerName="istio-proxy" containerID="cri-o://ef1639f46473a97e7e9800a67deac92528ca8dd3362c8fb309f33096e033ce39" gracePeriod=30
Oct 29 22:08:06.226686 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:06.226365    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"]
Oct 29 22:08:06.226686 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:06.226624    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc" podUID="e7cb54e0-6733-4774-be16-ccefc78e7d48" containerName="istio-proxy" containerID="cri-o://fd1c9b275988d036a78374eed25b008a0677694239c18c44c3b7384b85713e4b" gracePeriod=30
Oct 29 22:08:06.247065 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:06.247031    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:07.180831 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:07.180746    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:07.590963 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:07.590932    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:07.725252 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:07.716999    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:08.936057 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:08.936026    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:09.146578 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:09.146503    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:10.189966 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:10.189910    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:11.232790 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.232757    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:11.422762 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.422738    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:08:11.488577 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.486268    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ce2b92-3933-439a-b273-d313d80ecaf1-kube-api-access-lxbzm\" (UniqueName: \"kubernetes.io/projected/73ce2b92-3933-439a-b273-d313d80ecaf1-kube-api-access-lxbzm\") pod \"73ce2b92-3933-439a-b273-d313d80ecaf1\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") "
Oct 29 22:08:11.488577 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.486338    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-credential-socket\") pod \"73ce2b92-3933-439a-b273-d313d80ecaf1\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") "
Oct 29 22:08:11.488577 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.486378    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-envoy\") pod \"73ce2b92-3933-439a-b273-d313d80ecaf1\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") "
Oct 29 22:08:11.488577 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.486492    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ce2b92-3933-439a-b273-d313d80ecaf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ce2b92-3933-439a-b273-d313d80ecaf1-istiod-ca-cert\") pod \"73ce2b92-3933-439a-b273-d313d80ecaf1\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") "
Oct 29 22:08:11.488577 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.486562    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-data\") pod \"73ce2b92-3933-439a-b273-d313d80ecaf1\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") "
Oct 29 22:08:11.488577 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.486601    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-workload-certs\") pod \"73ce2b92-3933-439a-b273-d313d80ecaf1\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") "
Oct 29 22:08:11.488577 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.486629    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-podinfo\") pod \"73ce2b92-3933-439a-b273-d313d80ecaf1\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") "
Oct 29 22:08:11.488577 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.486662    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-workload-socket\") pod \"73ce2b92-3933-439a-b273-d313d80ecaf1\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") "
Oct 29 22:08:11.488577 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.486751    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-token\" (UniqueName: \"kubernetes.io/projected/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-token\") pod \"73ce2b92-3933-439a-b273-d313d80ecaf1\" (UID: \"73ce2b92-3933-439a-b273-d313d80ecaf1\") "
Oct 29 22:08:11.488577 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.487972    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ce2b92-3933-439a-b273-d313d80ecaf1-istiod-ca-cert" pod "73ce2b92-3933-439a-b273-d313d80ecaf1" (UID: "73ce2b92-3933-439a-b273-d313d80ecaf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:11.488577 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.488260    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-credential-socket" pod "73ce2b92-3933-439a-b273-d313d80ecaf1" (UID: "73ce2b92-3933-439a-b273-d313d80ecaf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.491100 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.491031    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-workload-socket" pod "73ce2b92-3933-439a-b273-d313d80ecaf1" (UID: "73ce2b92-3933-439a-b273-d313d80ecaf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.491100 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.490131    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-data" pod "73ce2b92-3933-439a-b273-d313d80ecaf1" (UID: "73ce2b92-3933-439a-b273-d313d80ecaf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.491365 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.491339    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-workload-certs" pod "73ce2b92-3933-439a-b273-d313d80ecaf1" (UID: "73ce2b92-3933-439a-b273-d313d80ecaf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.517690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.514907    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"]
Oct 29 22:08:11.517690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.516072    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c52c5564-6eef-4835-8f36-2f378dea76fb" containerName="revoke-keys"
Oct 29 22:08:11.517690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.516094    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52c5564-6eef-4835-8f36-2f378dea76fb" containerName="revoke-keys"
Oct 29 22:08:11.517690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.516115    4550 cpu_manager.go:414] "containerMap: removing container" podUID="73ce2b92-3933-439a-b273-d313d80ecaf1" containerName="istio-proxy"
Oct 29 22:08:11.517690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.516122    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ce2b92-3933-439a-b273-d313d80ecaf1" containerName="istio-proxy"
Oct 29 22:08:11.517690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.516388    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52c5564-6eef-4835-8f36-2f378dea76fb" containerName="revoke-keys"
Oct 29 22:08:11.517690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.516403    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ce2b92-3933-439a-b273-d313d80ecaf1" containerName="istio-proxy"
Oct 29 22:08:11.517690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.517343    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ce2b92-3933-439a-b273-d313d80ecaf1-kube-api-access-lxbzm" pod "73ce2b92-3933-439a-b273-d313d80ecaf1" (UID: "73ce2b92-3933-439a-b273-d313d80ecaf1"). InnerVolumeSpecName "kube-api-access-lxbzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:11.517690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.517441    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-podinfo" pod "73ce2b92-3933-439a-b273-d313d80ecaf1" (UID: "73ce2b92-3933-439a-b273-d313d80ecaf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:11.517690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.517498    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-envoy" pod "73ce2b92-3933-439a-b273-d313d80ecaf1" (UID: "73ce2b92-3933-439a-b273-d313d80ecaf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.517690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.517652    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-token" pod "73ce2b92-3933-439a-b273-d313d80ecaf1" (UID: "73ce2b92-3933-439a-b273-d313d80ecaf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:11.522451 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.522430    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.532270 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.532248    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"]
Oct 29 22:08:11.540930 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.540885    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9ae129-openshift-default-dockercfg-ltsf7\""
Oct 29 22:08:11.541892 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:08:11.541866    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:08:11.579928 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.579900    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:08:11.589103 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589077    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-podinfo\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.589223 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589118    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-envoy\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.589301 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589234    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-credential-socket\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.589357 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589327    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-token\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.589424 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589408    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpns5\" (UniqueName: \"kubernetes.io/projected/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-kube-api-access-mpns5\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.589478 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589438    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-workload-socket\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.589478 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589470    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-data\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.589602 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589589    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istiod-ca-cert\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.589727 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589714    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-workload-certs\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.589845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589834    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxbzm\" (UniqueName: \"kubernetes.io/projected/73ce2b92-3933-439a-b273-d313d80ecaf1-kube-api-access-lxbzm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.589881 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589848    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.589881 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589856    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.589881 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589866    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ce2b92-3933-439a-b273-d313d80ecaf1-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.589881 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589876    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.590003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589887    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.590003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589897    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.590003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589905    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ce2b92-3933-439a-b273-d313d80ecaf1-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.590003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.589913    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ce2b92-3933-439a-b273-d313d80ecaf1-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.643169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.643126    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"]
Oct 29 22:08:11.644068 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.644046    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e7cb54e0-6733-4774-be16-ccefc78e7d48" containerName="istio-proxy"
Oct 29 22:08:11.644068 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.644066    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cb54e0-6733-4774-be16-ccefc78e7d48" containerName="istio-proxy"
Oct 29 22:08:11.644320 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.644228    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cb54e0-6733-4774-be16-ccefc78e7d48" containerName="istio-proxy"
Oct 29 22:08:11.647964 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.647943    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"
Oct 29 22:08:11.666327 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.666271    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f363a3\""
Oct 29 22:08:11.668362 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.668340    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f363a3-dockercfg-m55vq\""
Oct 29 22:08:11.671538 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.671512    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"]
Oct 29 22:08:11.691209 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.691132    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-data\") pod \"e7cb54e0-6733-4774-be16-ccefc78e7d48\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") "
Oct 29 22:08:11.691337 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.691319    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7cb54e0-6733-4774-be16-ccefc78e7d48-kube-api-access-2m6qw\" (UniqueName: \"kubernetes.io/projected/e7cb54e0-6733-4774-be16-ccefc78e7d48-kube-api-access-2m6qw\") pod \"e7cb54e0-6733-4774-be16-ccefc78e7d48\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") "
Oct 29 22:08:11.691464 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.691437    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-data" pod "e7cb54e0-6733-4774-be16-ccefc78e7d48" (UID: "e7cb54e0-6733-4774-be16-ccefc78e7d48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.691544 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.691511    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-workload-certs\") pod \"e7cb54e0-6733-4774-be16-ccefc78e7d48\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") "
Oct 29 22:08:11.691677 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.691648    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-workload-socket\") pod \"e7cb54e0-6733-4774-be16-ccefc78e7d48\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") "
Oct 29 22:08:11.691880 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.691755    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-token\" (UniqueName: \"kubernetes.io/projected/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-token\") pod \"e7cb54e0-6733-4774-be16-ccefc78e7d48\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") "
Oct 29 22:08:11.691880 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.691795    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-podinfo\") pod \"e7cb54e0-6733-4774-be16-ccefc78e7d48\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") "
Oct 29 22:08:11.691880 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.691877    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7cb54e0-6733-4774-be16-ccefc78e7d48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7cb54e0-6733-4774-be16-ccefc78e7d48-istiod-ca-cert\") pod \"e7cb54e0-6733-4774-be16-ccefc78e7d48\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") "
Oct 29 22:08:11.692079 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.691928    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-workload-certs" pod "e7cb54e0-6733-4774-be16-ccefc78e7d48" (UID: "e7cb54e0-6733-4774-be16-ccefc78e7d48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.692079 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.691984    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-envoy\") pod \"e7cb54e0-6733-4774-be16-ccefc78e7d48\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") "
Oct 29 22:08:11.692079 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.692025    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-credential-socket\") pod \"e7cb54e0-6733-4774-be16-ccefc78e7d48\" (UID: \"e7cb54e0-6733-4774-be16-ccefc78e7d48\") "
Oct 29 22:08:11.692259 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.692235    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-workload-socket" pod "e7cb54e0-6733-4774-be16-ccefc78e7d48" (UID: "e7cb54e0-6733-4774-be16-ccefc78e7d48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.693423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.692385    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istiod-ca-cert\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.693423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.692501    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p74kd\" (UniqueName: \"kubernetes.io/projected/11f45a3b-d463-4335-b8f5-e43831a38ceb-kube-api-access-p74kd\") pod \"payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6\" (UID: \"11f45a3b-d463-4335-b8f5-e43831a38ceb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"
Oct 29 22:08:11.693423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.692509    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-credential-socket" pod "e7cb54e0-6733-4774-be16-ccefc78e7d48" (UID: "e7cb54e0-6733-4774-be16-ccefc78e7d48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.693423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.692678    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-workload-certs\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.693423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.692729    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f45a3b-d463-4335-b8f5-e43831a38ceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6\" (UID: \"11f45a3b-d463-4335-b8f5-e43831a38ceb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"
Oct 29 22:08:11.693423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.692809    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-podinfo\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.693423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.692890    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-envoy\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.693423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.692990    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-credential-socket\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.693423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.693035    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-token\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.693423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.693194    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-workload-certs\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.694048 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.693468    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-credential-socket\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.694048 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.693686    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cb54e0-6733-4774-be16-ccefc78e7d48-istiod-ca-cert" pod "e7cb54e0-6733-4774-be16-ccefc78e7d48" (UID: "e7cb54e0-6733-4774-be16-ccefc78e7d48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:11.694048 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.693724    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpns5\" (UniqueName: \"kubernetes.io/projected/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-kube-api-access-mpns5\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.694048 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.693747    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istiod-ca-cert\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.694048 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.693769    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-workload-socket\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.694048 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.693861    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-data\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.694048 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.694019    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-workload-socket\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.694448 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.694116    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.694448 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.694154    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7cb54e0-6733-4774-be16-ccefc78e7d48-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.694448 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.694171    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.694448 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.694186    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.694448 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.694201    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.694448 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.694219    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-data\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.695725 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.695696    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-envoy\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.695830 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.695719    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-envoy" pod "e7cb54e0-6733-4774-be16-ccefc78e7d48" (UID: "e7cb54e0-6733-4774-be16-ccefc78e7d48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.697233 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.697199    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-token" pod "e7cb54e0-6733-4774-be16-ccefc78e7d48" (UID: "e7cb54e0-6733-4774-be16-ccefc78e7d48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:11.697321 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.697289    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-podinfo" pod "e7cb54e0-6733-4774-be16-ccefc78e7d48" (UID: "e7cb54e0-6733-4774-be16-ccefc78e7d48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:11.697514 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.697489    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-podinfo\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.704073 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.704048    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cb54e0-6733-4774-be16-ccefc78e7d48-kube-api-access-2m6qw" pod "e7cb54e0-6733-4774-be16-ccefc78e7d48" (UID: "e7cb54e0-6733-4774-be16-ccefc78e7d48"). InnerVolumeSpecName "kube-api-access-2m6qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:11.719248 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.719226    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-token\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.719342 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.719248    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpns5\" (UniqueName: \"kubernetes.io/projected/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-kube-api-access-mpns5\") pod \"e2e-api-a-9ae129-openshift-default-65fc746555-g28qc\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") " pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.795826 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.795749    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p74kd\" (UniqueName: \"kubernetes.io/projected/11f45a3b-d463-4335-b8f5-e43831a38ceb-kube-api-access-p74kd\") pod \"payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6\" (UID: \"11f45a3b-d463-4335-b8f5-e43831a38ceb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"
Oct 29 22:08:11.795942 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.795876    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f45a3b-d463-4335-b8f5-e43831a38ceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6\" (UID: \"11f45a3b-d463-4335-b8f5-e43831a38ceb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"
Oct 29 22:08:11.796078 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.796059    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m6qw\" (UniqueName: \"kubernetes.io/projected/e7cb54e0-6733-4774-be16-ccefc78e7d48-kube-api-access-2m6qw\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.796124 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.796085    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.796124 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.796100    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.796124 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.796113    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7cb54e0-6733-4774-be16-ccefc78e7d48-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.796615 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.796595    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f45a3b-d463-4335-b8f5-e43831a38ceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6\" (UID: \"11f45a3b-d463-4335-b8f5-e43831a38ceb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"
Oct 29 22:08:11.820696 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.820656    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p74kd\" (UniqueName: \"kubernetes.io/projected/11f45a3b-d463-4335-b8f5-e43831a38ceb-kube-api-access-p74kd\") pod \"payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6\" (UID: \"11f45a3b-d463-4335-b8f5-e43831a38ceb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"
Oct 29 22:08:11.848852 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.848824    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:11.969118 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.969088    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"
Oct 29 22:08:11.978504 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:11.978472    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"]
Oct 29 22:08:12.014257 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.014233    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"]
Oct 29 22:08:12.118396 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.118353    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:12.178505 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.178459    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"]
Oct 29 22:08:12.181399 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:08:12.181365    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec85ac8_9f07_402f_98d3_cb8b5f5fe581.slice/crio-3ad44b053879c8b12c892ec9602a4de46d5b646c039e5e242fb528270ea3c81e WatchSource:0}: Error finding container 3ad44b053879c8b12c892ec9602a4de46d5b646c039e5e242fb528270ea3c81e: Status 404 returned error can't find the container with id 3ad44b053879c8b12c892ec9602a4de46d5b646c039e5e242fb528270ea3c81e
Oct 29 22:08:12.184897 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.184860    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:08:12.184976 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.184954    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:08:12.185020 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.185005    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:08:12.240138 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.240105    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:12.255434 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.255399    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" event={"ID":"aec85ac8-9f07-402f-98d3-cb8b5f5fe581","Type":"ContainerStarted","Data":"3ad44b053879c8b12c892ec9602a4de46d5b646c039e5e242fb528270ea3c81e"}
Oct 29 22:08:12.257284 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.257251    4550 generic.go:356] "Generic (PLEG): container finished" podID="73ce2b92-3933-439a-b273-d313d80ecaf1" containerID="ef1639f46473a97e7e9800a67deac92528ca8dd3362c8fb309f33096e033ce39" exitCode=0
Oct 29 22:08:12.257415 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.257399    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9" event={"ID":"73ce2b92-3933-439a-b273-d313d80ecaf1","Type":"ContainerDied","Data":"ef1639f46473a97e7e9800a67deac92528ca8dd3362c8fb309f33096e033ce39"}
Oct 29 22:08:12.257485 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.257445    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9" event={"ID":"73ce2b92-3933-439a-b273-d313d80ecaf1","Type":"ContainerDied","Data":"250a813c73cc13add8be1c21b319b442acd1d751a0d43f0cf0ac7a7e3147ffc1"}
Oct 29 22:08:12.257485 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.257474    4550 scope.go:122] "RemoveContainer" containerID="ef1639f46473a97e7e9800a67deac92528ca8dd3362c8fb309f33096e033ce39"
Oct 29 22:08:12.257643 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.257631    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"
Oct 29 22:08:12.260557 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.260518    4550 generic.go:356] "Generic (PLEG): container finished" podID="e7cb54e0-6733-4774-be16-ccefc78e7d48" containerID="fd1c9b275988d036a78374eed25b008a0677694239c18c44c3b7384b85713e4b" exitCode=0
Oct 29 22:08:12.260663 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.260575    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"
Oct 29 22:08:12.260760 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.260573    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc" event={"ID":"e7cb54e0-6733-4774-be16-ccefc78e7d48","Type":"ContainerDied","Data":"fd1c9b275988d036a78374eed25b008a0677694239c18c44c3b7384b85713e4b"}
Oct 29 22:08:12.260815 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.260786    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc" event={"ID":"e7cb54e0-6733-4774-be16-ccefc78e7d48","Type":"ContainerDied","Data":"91faa73ca86cf6ca7cfb0253169b523a92c4f2a9fc2433b9997aa3fa7c23a583"}
Oct 29 22:08:12.292113 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.292088    4550 scope.go:122] "RemoveContainer" containerID="ef1639f46473a97e7e9800a67deac92528ca8dd3362c8fb309f33096e033ce39"
Oct 29 22:08:12.294295 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.294266    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"]
Oct 29 22:08:12.294398 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:08:12.294287    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef1639f46473a97e7e9800a67deac92528ca8dd3362c8fb309f33096e033ce39\": container with ID starting with ef1639f46473a97e7e9800a67deac92528ca8dd3362c8fb309f33096e033ce39 not found: ID does not exist" containerID="ef1639f46473a97e7e9800a67deac92528ca8dd3362c8fb309f33096e033ce39"
Oct 29 22:08:12.294398 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.294321    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1639f46473a97e7e9800a67deac92528ca8dd3362c8fb309f33096e033ce39"} err="failed to get container status \"ef1639f46473a97e7e9800a67deac92528ca8dd3362c8fb309f33096e033ce39\": rpc error: code = NotFound desc = could not find container \"ef1639f46473a97e7e9800a67deac92528ca8dd3362c8fb309f33096e033ce39\": container with ID starting with ef1639f46473a97e7e9800a67deac92528ca8dd3362c8fb309f33096e033ce39 not found: ID does not exist"
Oct 29 22:08:12.294398 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.294340    4550 scope.go:122] "RemoveContainer" containerID="fd1c9b275988d036a78374eed25b008a0677694239c18c44c3b7384b85713e4b"
Oct 29 22:08:12.296847 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:08:12.296819    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f45a3b_d463_4335_b8f5_e43831a38ceb.slice/crio-7afb834208f284131d731ff80bff77a154d4dbc3fadfd8714beb8db1249aeaa7 WatchSource:0}: Error finding container 7afb834208f284131d731ff80bff77a154d4dbc3fadfd8714beb8db1249aeaa7: Status 404 returned error can't find the container with id 7afb834208f284131d731ff80bff77a154d4dbc3fadfd8714beb8db1249aeaa7
Oct 29 22:08:12.329718 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.329685    4550 scope.go:122] "RemoveContainer" containerID="fd1c9b275988d036a78374eed25b008a0677694239c18c44c3b7384b85713e4b"
Oct 29 22:08:12.330323 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:08:12.330292    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd1c9b275988d036a78374eed25b008a0677694239c18c44c3b7384b85713e4b\": container with ID starting with fd1c9b275988d036a78374eed25b008a0677694239c18c44c3b7384b85713e4b not found: ID does not exist" containerID="fd1c9b275988d036a78374eed25b008a0677694239c18c44c3b7384b85713e4b"
Oct 29 22:08:12.330497 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.330464    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd1c9b275988d036a78374eed25b008a0677694239c18c44c3b7384b85713e4b"} err="failed to get container status \"fd1c9b275988d036a78374eed25b008a0677694239c18c44c3b7384b85713e4b\": rpc error: code = NotFound desc = could not find container \"fd1c9b275988d036a78374eed25b008a0677694239c18c44c3b7384b85713e4b\": container with ID starting with fd1c9b275988d036a78374eed25b008a0677694239c18c44c3b7384b85713e4b not found: ID does not exist"
Oct 29 22:08:12.393494 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.390165    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:12.443405 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.441782    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"]
Oct 29 22:08:12.473742 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.473698    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tp8mc"]
Oct 29 22:08:12.477925 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.477853    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"]
Oct 29 22:08:12.485317 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.485198    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-tmcw9"]
Oct 29 22:08:12.724164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:12.722618    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:13.271322 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:13.271267    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6" event={"ID":"11f45a3b-d463-4335-b8f5-e43831a38ceb","Type":"ContainerStarted","Data":"7afb834208f284131d731ff80bff77a154d4dbc3fadfd8714beb8db1249aeaa7"}
Oct 29 22:08:13.273403 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:13.273287    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" event={"ID":"aec85ac8-9f07-402f-98d3-cb8b5f5fe581","Type":"ContainerStarted","Data":"1b27f106dfc515e505cf13888d762b8646cb0fb0d18481889c1465b92b5398c0"}
Oct 29 22:08:13.352028 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:13.351972    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podStartSLOduration=6.35195754 podStartE2EDuration="6.35195754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:13.335752829 +0000 UTC m=+1309.492878370" watchObservedRunningTime="2026-10-29 22:08:13.35195754 +0000 UTC m=+1309.509083038"
Oct 29 22:08:13.849871 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:13.849833    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:13.851276 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:13.851251    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:08:13.851360 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:13.851301    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:08:13.991881 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:13.991799    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ce2b92-3933-439a-b273-d313d80ecaf1" path="/var/lib/kubelet/pods/73ce2b92-3933-439a-b273-d313d80ecaf1/volumes"
Oct 29 22:08:13.992349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:13.992331    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7cb54e0-6733-4774-be16-ccefc78e7d48" path="/var/lib/kubelet/pods/e7cb54e0-6733-4774-be16-ccefc78e7d48/volumes"
Oct 29 22:08:14.129943 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:14.129909    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:14.290320 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:14.290235    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6" event={"ID":"11f45a3b-d463-4335-b8f5-e43831a38ceb","Type":"ContainerStarted","Data":"2f26ba83c235b86747d7f1eb4f3ba6cfde97c0b647e2055fe178203196b1321f"}
Oct 29 22:08:14.290698 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:14.290329    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"
Oct 29 22:08:14.338333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:14.338286    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6" podStartSLOduration=4.001080518 podStartE2EDuration="5.338272505s" totalImagesPullingTime="1.337191987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:14.334381726 +0000 UTC m=+1310.491507225" watchObservedRunningTime="2026-10-29 22:08:14.338272505 +0000 UTC m=+1310.495398003"
Oct 29 22:08:14.850023 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:14.849989    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:08:14.850211 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:14.850051    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:08:15.689868 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:15.689838    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:15.811156 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:15.811113    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:15.849993 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:15.849959    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:08:15.850227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:15.850022    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:08:16.460728 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:16.459941    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:16.849874 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:16.849840    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:08:16.850310 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:16.849895    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:08:16.979001 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:16.978967    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq" podUID="f061ec29-6d39-43d9-93ea-de6efb4a6489" containerName="payload-processing" containerID="cri-o://f78516fa9673215b6a174da2a012ac130f30c890196675b8f9928f6806cb664d" gracePeriod=25
Oct 29 22:08:17.014907 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.014876    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp" podUID="2625bcf3-456a-4112-9c40-d42f7da5cfda" containerName="payload-pre-processing" containerID="cri-o://7f3131fd0479d0b49f5d2d50039bb0ce6a3d58009c2177a103a68401342ab7bd" gracePeriod=25
Oct 29 22:08:17.317437 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.317400    4550 generic.go:356] "Generic (PLEG): container finished" podID="f061ec29-6d39-43d9-93ea-de6efb4a6489" containerID="f78516fa9673215b6a174da2a012ac130f30c890196675b8f9928f6806cb664d" exitCode=0
Oct 29 22:08:17.317606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.317454    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq" event={"ID":"f061ec29-6d39-43d9-93ea-de6efb4a6489","Type":"ContainerDied","Data":"f78516fa9673215b6a174da2a012ac130f30c890196675b8f9928f6806cb664d"}
Oct 29 22:08:17.317606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.317528    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq" event={"ID":"f061ec29-6d39-43d9-93ea-de6efb4a6489","Type":"ContainerDied","Data":"dcd7d910501255e3fe765e7a0e14656c25f137ba38e8ed9c76314b4095a777d2"}
Oct 29 22:08:17.317606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.317571    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcd7d910501255e3fe765e7a0e14656c25f137ba38e8ed9c76314b4095a777d2"
Oct 29 22:08:17.319753 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.319720    4550 generic.go:356] "Generic (PLEG): container finished" podID="2625bcf3-456a-4112-9c40-d42f7da5cfda" containerID="7f3131fd0479d0b49f5d2d50039bb0ce6a3d58009c2177a103a68401342ab7bd" exitCode=0
Oct 29 22:08:17.319910 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.319821    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp" event={"ID":"2625bcf3-456a-4112-9c40-d42f7da5cfda","Type":"ContainerDied","Data":"7f3131fd0479d0b49f5d2d50039bb0ce6a3d58009c2177a103a68401342ab7bd"}
Oct 29 22:08:17.341426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.341402    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"
Oct 29 22:08:17.385904 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.385872    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:17.393099 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.391578    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"
Oct 29 22:08:17.397768 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.397734    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f061ec29-6d39-43d9-93ea-de6efb4a6489-kube-api-access-qxszn\" (UniqueName: \"kubernetes.io/projected/f061ec29-6d39-43d9-93ea-de6efb4a6489-kube-api-access-qxszn\") pod \"f061ec29-6d39-43d9-93ea-de6efb4a6489\" (UID: \"f061ec29-6d39-43d9-93ea-de6efb4a6489\") "
Oct 29 22:08:17.397899 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.397800    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f061ec29-6d39-43d9-93ea-de6efb4a6489-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f061ec29-6d39-43d9-93ea-de6efb4a6489-plugins-config-volume\") pod \"f061ec29-6d39-43d9-93ea-de6efb4a6489\" (UID: \"f061ec29-6d39-43d9-93ea-de6efb4a6489\") "
Oct 29 22:08:17.399401 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.399313    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f061ec29-6d39-43d9-93ea-de6efb4a6489-plugins-config-volume" pod "f061ec29-6d39-43d9-93ea-de6efb4a6489" (UID: "f061ec29-6d39-43d9-93ea-de6efb4a6489"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:17.402230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.402195    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f061ec29-6d39-43d9-93ea-de6efb4a6489-kube-api-access-qxszn" pod "f061ec29-6d39-43d9-93ea-de6efb4a6489" (UID: "f061ec29-6d39-43d9-93ea-de6efb4a6489"). InnerVolumeSpecName "kube-api-access-qxszn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:17.499751 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.499713    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2625bcf3-456a-4112-9c40-d42f7da5cfda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2625bcf3-456a-4112-9c40-d42f7da5cfda-plugins-config-volume\") pod \"2625bcf3-456a-4112-9c40-d42f7da5cfda\" (UID: \"2625bcf3-456a-4112-9c40-d42f7da5cfda\") "
Oct 29 22:08:17.499928 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.499889    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2625bcf3-456a-4112-9c40-d42f7da5cfda-kube-api-access-5m7v2\" (UniqueName: \"kubernetes.io/projected/2625bcf3-456a-4112-9c40-d42f7da5cfda-kube-api-access-5m7v2\") pod \"2625bcf3-456a-4112-9c40-d42f7da5cfda\" (UID: \"2625bcf3-456a-4112-9c40-d42f7da5cfda\") "
Oct 29 22:08:17.500240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.500209    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2625bcf3-456a-4112-9c40-d42f7da5cfda-plugins-config-volume" pod "2625bcf3-456a-4112-9c40-d42f7da5cfda" (UID: "2625bcf3-456a-4112-9c40-d42f7da5cfda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:17.500707 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.500687    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxszn\" (UniqueName: \"kubernetes.io/projected/f061ec29-6d39-43d9-93ea-de6efb4a6489-kube-api-access-qxszn\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:17.500803 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.500713    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f061ec29-6d39-43d9-93ea-de6efb4a6489-plugins-config-volume\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:17.500803 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.500727    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2625bcf3-456a-4112-9c40-d42f7da5cfda-plugins-config-volume\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:17.503362 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.503332    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2625bcf3-456a-4112-9c40-d42f7da5cfda-kube-api-access-5m7v2" pod "2625bcf3-456a-4112-9c40-d42f7da5cfda" (UID: "2625bcf3-456a-4112-9c40-d42f7da5cfda"). InnerVolumeSpecName "kube-api-access-5m7v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:17.509560 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.509533    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:17.624454 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.624351    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m7v2\" (UniqueName: \"kubernetes.io/projected/2625bcf3-456a-4112-9c40-d42f7da5cfda-kube-api-access-5m7v2\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:17.850904 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.850202    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:08:17.850904 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:17.850258    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:08:18.329097 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:18.329065    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"
Oct 29 22:08:18.329097 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:18.329068    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp" event={"ID":"2625bcf3-456a-4112-9c40-d42f7da5cfda","Type":"ContainerDied","Data":"9433bc09be6627e0ea38a8cfaa12445f7323f8210e54d7c14dc3b29347269ce6"}
Oct 29 22:08:18.329383 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:18.329166    4550 scope.go:122] "RemoveContainer" containerID="7f3131fd0479d0b49f5d2d50039bb0ce6a3d58009c2177a103a68401342ab7bd"
Oct 29 22:08:18.329473 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:18.329444    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"
Oct 29 22:08:18.386827 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:18.386802    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"]
Oct 29 22:08:18.393927 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:18.393904    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a26f29-5f5cb9764-4r6zq"]
Oct 29 22:08:18.428335 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:18.428310    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"]
Oct 29 22:08:18.431815 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:18.431791    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a26f29-6d695f674c-mncsp"]
Oct 29 22:08:18.474246 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:18.471257    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:18.849734 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:18.849697    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:08:18.849909 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:18.849751    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:08:19.849914 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:19.849879    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:08:19.850462 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:19.849944    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:08:20.099119 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:20.083929    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2625bcf3-456a-4112-9c40-d42f7da5cfda" path="/var/lib/kubelet/pods/2625bcf3-456a-4112-9c40-d42f7da5cfda/volumes"
Oct 29 22:08:20.099119 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:20.084875    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f061ec29-6d39-43d9-93ea-de6efb4a6489" path="/var/lib/kubelet/pods/f061ec29-6d39-43d9-93ea-de6efb4a6489/volumes"
Oct 29 22:08:20.411898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:20.411856    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:20.849174 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:20.849126    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:08:20.849353 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:20.849201    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:08:21.173074 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:21.172925    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:21.309244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:21.309205    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:21.852166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:21.851544    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:08:21.852166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:21.851630    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:08:22.497106 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:22.497069    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:22.853792 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:22.853762    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:22.854207 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:22.854188    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:22.854976 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:22.854954    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:08:24.270333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:24.270294    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:25.229615 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:25.229578    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:25.299207 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:25.299157    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"
Oct 29 22:08:25.350069 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:25.350038    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:26.390156 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:26.390117    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:27.736408 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:27.736374    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:28.261333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:28.261276    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:28.415463 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:28.415432    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:28.552355 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:28.552271    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:28.942903 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:28.942868    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:29.165107 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:29.165072    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:29.312766 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:29.312679    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:29.477380 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:29.477330    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8jrxh"]
Oct 29 22:08:29.477994 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:29.477943    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/odh-model-controller-769875db56-8jrxh" podUID="b10e7c81-07e3-4c64-8433-ee31a9a26fbb" containerName="manager" containerID="cri-o://3609780eb10f2dbf22d9349a6169ca0e9bf5fe5daa938585d38ad3dfa4fec19a" gracePeriod=10
Oct 29 22:08:29.831992 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:29.831968    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-8jrxh"
Oct 29 22:08:29.834222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:29.834195    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:29.967806 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:29.967708    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-cert\" (UniqueName: \"kubernetes.io/secret/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-cert\") pod \"b10e7c81-07e3-4c64-8433-ee31a9a26fbb\" (UID: \"b10e7c81-07e3-4c64-8433-ee31a9a26fbb\") "
Oct 29 22:08:29.968347 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:29.967900    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-metrics-certs\") pod \"b10e7c81-07e3-4c64-8433-ee31a9a26fbb\" (UID: \"b10e7c81-07e3-4c64-8433-ee31a9a26fbb\") "
Oct 29 22:08:29.968347 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:29.968040    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-kube-api-access-djjlh\" (UniqueName: \"kubernetes.io/projected/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-kube-api-access-djjlh\") pod \"b10e7c81-07e3-4c64-8433-ee31a9a26fbb\" (UID: \"b10e7c81-07e3-4c64-8433-ee31a9a26fbb\") "
Oct 29 22:08:29.971407 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:29.971381    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-kube-api-access-djjlh" pod "b10e7c81-07e3-4c64-8433-ee31a9a26fbb" (UID: "b10e7c81-07e3-4c64-8433-ee31a9a26fbb"). InnerVolumeSpecName "kube-api-access-djjlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:29.971660 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:29.971638    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-cert" pod "b10e7c81-07e3-4c64-8433-ee31a9a26fbb" (UID: "b10e7c81-07e3-4c64-8433-ee31a9a26fbb"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:08:29.973181 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:29.973117    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-metrics-certs" pod "b10e7c81-07e3-4c64-8433-ee31a9a26fbb" (UID: "b10e7c81-07e3-4c64-8433-ee31a9a26fbb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:08:30.072119 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:30.072084    4550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-metrics-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:30.072119 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:30.072120    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djjlh\" (UniqueName: \"kubernetes.io/projected/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-kube-api-access-djjlh\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:30.072370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:30.072137    4550 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b10e7c81-07e3-4c64-8433-ee31a9a26fbb-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:30.447022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:30.446981    4550 generic.go:356] "Generic (PLEG): container finished" podID="b10e7c81-07e3-4c64-8433-ee31a9a26fbb" containerID="3609780eb10f2dbf22d9349a6169ca0e9bf5fe5daa938585d38ad3dfa4fec19a" exitCode=0
Oct 29 22:08:30.447249 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:30.447078    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-8jrxh"
Oct 29 22:08:30.447249 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:30.447078    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-8jrxh" event={"ID":"b10e7c81-07e3-4c64-8433-ee31a9a26fbb","Type":"ContainerDied","Data":"3609780eb10f2dbf22d9349a6169ca0e9bf5fe5daa938585d38ad3dfa4fec19a"}
Oct 29 22:08:30.447249 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:30.447230    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-8jrxh" event={"ID":"b10e7c81-07e3-4c64-8433-ee31a9a26fbb","Type":"ContainerDied","Data":"1501df7ed26c06b9a6a271474e463813bda7cd5c85b6402cf8f862345691a7a7"}
Oct 29 22:08:30.447406 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:30.447264    4550 scope.go:122] "RemoveContainer" containerID="3609780eb10f2dbf22d9349a6169ca0e9bf5fe5daa938585d38ad3dfa4fec19a"
Oct 29 22:08:30.461599 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:30.461573    4550 scope.go:122] "RemoveContainer" containerID="3609780eb10f2dbf22d9349a6169ca0e9bf5fe5daa938585d38ad3dfa4fec19a"
Oct 29 22:08:30.461910 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:08:30.461884    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3609780eb10f2dbf22d9349a6169ca0e9bf5fe5daa938585d38ad3dfa4fec19a\": container with ID starting with 3609780eb10f2dbf22d9349a6169ca0e9bf5fe5daa938585d38ad3dfa4fec19a not found: ID does not exist" containerID="3609780eb10f2dbf22d9349a6169ca0e9bf5fe5daa938585d38ad3dfa4fec19a"
Oct 29 22:08:30.462022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:30.461923    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3609780eb10f2dbf22d9349a6169ca0e9bf5fe5daa938585d38ad3dfa4fec19a"} err="failed to get container status \"3609780eb10f2dbf22d9349a6169ca0e9bf5fe5daa938585d38ad3dfa4fec19a\": rpc error: code = NotFound desc = could not find container \"3609780eb10f2dbf22d9349a6169ca0e9bf5fe5daa938585d38ad3dfa4fec19a\": container with ID starting with 3609780eb10f2dbf22d9349a6169ca0e9bf5fe5daa938585d38ad3dfa4fec19a not found: ID does not exist"
Oct 29 22:08:30.509995 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:30.509961    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8jrxh"]
Oct 29 22:08:30.526734 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:30.526701    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8jrxh"]
Oct 29 22:08:31.862214 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:31.862178    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:31.998042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:31.998006    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b10e7c81-07e3-4c64-8433-ee31a9a26fbb" path="/var/lib/kubelet/pods/b10e7c81-07e3-4c64-8433-ee31a9a26fbb/volumes"
Oct 29 22:08:32.770687 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.770655    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-dq4xb"]
Oct 29 22:08:32.771366 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.771351    4550 cpu_manager.go:414] "containerMap: removing container" podUID="f061ec29-6d39-43d9-93ea-de6efb4a6489" containerName="payload-processing"
Oct 29 22:08:32.771427 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.771367    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f061ec29-6d39-43d9-93ea-de6efb4a6489" containerName="payload-processing"
Oct 29 22:08:32.771427 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.771400    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b10e7c81-07e3-4c64-8433-ee31a9a26fbb" containerName="manager"
Oct 29 22:08:32.771427 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.771407    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10e7c81-07e3-4c64-8433-ee31a9a26fbb" containerName="manager"
Oct 29 22:08:32.771427 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.771422    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2625bcf3-456a-4112-9c40-d42f7da5cfda" containerName="payload-pre-processing"
Oct 29 22:08:32.771567 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.771428    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2625bcf3-456a-4112-9c40-d42f7da5cfda" containerName="payload-pre-processing"
Oct 29 22:08:32.771567 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.771530    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10e7c81-07e3-4c64-8433-ee31a9a26fbb" containerName="manager"
Oct 29 22:08:32.771567 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.771541    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2625bcf3-456a-4112-9c40-d42f7da5cfda" containerName="payload-pre-processing"
Oct 29 22:08:32.771567 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.771555    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="f061ec29-6d39-43d9-93ea-de6efb4a6489" containerName="payload-processing"
Oct 29 22:08:32.776888 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.776867    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-dq4xb"
Oct 29 22:08:32.808544 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.808507    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-lzrxx\""
Oct 29 22:08:32.808744 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.808723    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf98d9e-00a0-4795-a5e6-f494f873fd0d-metrics-certs\") pod \"odh-model-controller-769875db56-dq4xb\" (UID: \"fdf98d9e-00a0-4795-a5e6-f494f873fd0d\") " pod="opendatahub/odh-model-controller-769875db56-dq4xb"
Oct 29 22:08:32.808820 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.808747    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 29 22:08:32.808820 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.808792    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdf98d9e-00a0-4795-a5e6-f494f873fd0d-cert\") pod \"odh-model-controller-769875db56-dq4xb\" (UID: \"fdf98d9e-00a0-4795-a5e6-f494f873fd0d\") " pod="opendatahub/odh-model-controller-769875db56-dq4xb"
Oct 29 22:08:32.808820 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.808811    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 29 22:08:32.808981 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.808916    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xkpz\" (UniqueName: \"kubernetes.io/projected/fdf98d9e-00a0-4795-a5e6-f494f873fd0d-kube-api-access-8xkpz\") pod \"odh-model-controller-769875db56-dq4xb\" (UID: \"fdf98d9e-00a0-4795-a5e6-f494f873fd0d\") " pod="opendatahub/odh-model-controller-769875db56-dq4xb"
Oct 29 22:08:32.813825 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.813797    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-dq4xb"]
Oct 29 22:08:32.910941 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.910906    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xkpz\" (UniqueName: \"kubernetes.io/projected/fdf98d9e-00a0-4795-a5e6-f494f873fd0d-kube-api-access-8xkpz\") pod \"odh-model-controller-769875db56-dq4xb\" (UID: \"fdf98d9e-00a0-4795-a5e6-f494f873fd0d\") " pod="opendatahub/odh-model-controller-769875db56-dq4xb"
Oct 29 22:08:32.911475 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.911118    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf98d9e-00a0-4795-a5e6-f494f873fd0d-metrics-certs\") pod \"odh-model-controller-769875db56-dq4xb\" (UID: \"fdf98d9e-00a0-4795-a5e6-f494f873fd0d\") " pod="opendatahub/odh-model-controller-769875db56-dq4xb"
Oct 29 22:08:32.911475 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.911183    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdf98d9e-00a0-4795-a5e6-f494f873fd0d-cert\") pod \"odh-model-controller-769875db56-dq4xb\" (UID: \"fdf98d9e-00a0-4795-a5e6-f494f873fd0d\") " pod="opendatahub/odh-model-controller-769875db56-dq4xb"
Oct 29 22:08:32.913748 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.913722    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf98d9e-00a0-4795-a5e6-f494f873fd0d-metrics-certs\") pod \"odh-model-controller-769875db56-dq4xb\" (UID: \"fdf98d9e-00a0-4795-a5e6-f494f873fd0d\") " pod="opendatahub/odh-model-controller-769875db56-dq4xb"
Oct 29 22:08:32.913964 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.913729    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdf98d9e-00a0-4795-a5e6-f494f873fd0d-cert\") pod \"odh-model-controller-769875db56-dq4xb\" (UID: \"fdf98d9e-00a0-4795-a5e6-f494f873fd0d\") " pod="opendatahub/odh-model-controller-769875db56-dq4xb"
Oct 29 22:08:32.936197 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.936132    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xkpz\" (UniqueName: \"kubernetes.io/projected/fdf98d9e-00a0-4795-a5e6-f494f873fd0d-kube-api-access-8xkpz\") pod \"odh-model-controller-769875db56-dq4xb\" (UID: \"fdf98d9e-00a0-4795-a5e6-f494f873fd0d\") " pod="opendatahub/odh-model-controller-769875db56-dq4xb"
Oct 29 22:08:32.981154 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:32.981098    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:33.093218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:33.093183    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-dq4xb"
Oct 29 22:08:33.151307 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:33.147650    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:33.407048 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:33.406969    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-dq4xb"]
Oct 29 22:08:33.474439 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:33.474403    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-dq4xb" event={"ID":"fdf98d9e-00a0-4795-a5e6-f494f873fd0d","Type":"ContainerStarted","Data":"ba74c82d98e1c88e805fa7caec442d1ba03e83d4e6f4bf0ed6560901edbab27b"}
Oct 29 22:08:34.487542 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:34.487507    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-dq4xb" event={"ID":"fdf98d9e-00a0-4795-a5e6-f494f873fd0d","Type":"ContainerStarted","Data":"4170f1eb7f5dd9333c37acf381a5f36bfb35dcb484eba83b2bf814e7da30bd1b"}
Oct 29 22:08:34.487979 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:34.487733    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-dq4xb"
Oct 29 22:08:34.542885 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:34.542835    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-dq4xb" podStartSLOduration=2.130976455 podStartE2EDuration="2.542820247s" totalImagesPullingTime="411.843792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:34.540249061 +0000 UTC m=+1330.697374559" watchObservedRunningTime="2026-10-29 22:08:34.542820247 +0000 UTC m=+1330.699945766"
Oct 29 22:08:34.901212 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:34.901177    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:35.034637 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:35.034598    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:36.728943 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:36.728906    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:38.376335 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:38.376299    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:39.240578 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:39.240545    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:39.360815 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:39.360777    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:39.574678 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:39.574644    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"]
Oct 29 22:08:39.575477 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:39.575436    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz" podUID="606c44a7-c952-49b1-b5eb-6f527d149c02" containerName="istio-proxy" containerID="cri-o://6d8d8a9ccd5ab2288670821dd9e99c207efbcee77811449a1bca29335d15aa73" gracePeriod=30
Oct 29 22:08:39.588602 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:39.588567    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"]
Oct 29 22:08:39.588840 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:39.588797    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55" podUID="06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" containerName="istio-proxy" containerID="cri-o://b862f9c98dbe483f54d78b0d306fff152e55854b8402cc84177057b361f195b7" gracePeriod=30
Oct 29 22:08:39.624195 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:39.619940    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:40.430242 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:40.430209    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:41.496181 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:08:41.496120    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:08:41.961124 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:41.961080    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:43.088330 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:43.088297    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:44.882197 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:44.882121    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:08:44.914321 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:44.914297    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:08:45.024020 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.023989    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-data\") pod \"606c44a7-c952-49b1-b5eb-6f527d149c02\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") "
Oct 29 22:08:45.024225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024032    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-token\" (UniqueName: \"kubernetes.io/projected/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-token\") pod \"606c44a7-c952-49b1-b5eb-6f527d149c02\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") "
Oct 29 22:08:45.024225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024064    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-podinfo\") pod \"606c44a7-c952-49b1-b5eb-6f527d149c02\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") "
Oct 29 22:08:45.024225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024096    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-credential-socket\") pod \"606c44a7-c952-49b1-b5eb-6f527d149c02\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") "
Oct 29 22:08:45.024225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024126    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-kube-api-access-f9q95\" (UniqueName: \"kubernetes.io/projected/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-kube-api-access-f9q95\") pod \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") "
Oct 29 22:08:45.024225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024202    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-envoy\") pod \"606c44a7-c952-49b1-b5eb-6f527d149c02\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") "
Oct 29 22:08:45.024534 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024275    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606c44a7-c952-49b1-b5eb-6f527d149c02-kube-api-access-5fbtf\" (UniqueName: \"kubernetes.io/projected/606c44a7-c952-49b1-b5eb-6f527d149c02-kube-api-access-5fbtf\") pod \"606c44a7-c952-49b1-b5eb-6f527d149c02\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") "
Oct 29 22:08:45.024534 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024308    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-data" pod "606c44a7-c952-49b1-b5eb-6f527d149c02" (UID: "606c44a7-c952-49b1-b5eb-6f527d149c02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.024534 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024414    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-workload-socket\") pod \"606c44a7-c952-49b1-b5eb-6f527d149c02\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") "
Oct 29 22:08:45.024707 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024555    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-workload-socket\") pod \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") "
Oct 29 22:08:45.024707 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024609    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-workload-certs\") pod \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") "
Oct 29 22:08:45.024707 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024639    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-podinfo\") pod \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") "
Oct 29 22:08:45.024707 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024669    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-workload-certs\") pod \"606c44a7-c952-49b1-b5eb-6f527d149c02\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") "
Oct 29 22:08:45.024912 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024714    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-credential-socket\") pod \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") "
Oct 29 22:08:45.024912 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024792    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-data\") pod \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") "
Oct 29 22:08:45.024912 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024844    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-envoy\") pod \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") "
Oct 29 22:08:45.025059 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024941    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-token\" (UniqueName: \"kubernetes.io/projected/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-token\") pod \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") "
Oct 29 22:08:45.025059 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.024967    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-credential-socket" pod "606c44a7-c952-49b1-b5eb-6f527d149c02" (UID: "606c44a7-c952-49b1-b5eb-6f527d149c02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.025059 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.025008    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istiod-ca-cert\") pod \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\" (UID: \"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757\") "
Oct 29 22:08:45.025059 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.025035    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/606c44a7-c952-49b1-b5eb-6f527d149c02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606c44a7-c952-49b1-b5eb-6f527d149c02-istiod-ca-cert\") pod \"606c44a7-c952-49b1-b5eb-6f527d149c02\" (UID: \"606c44a7-c952-49b1-b5eb-6f527d149c02\") "
Oct 29 22:08:45.025312 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.025204    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-workload-certs" pod "606c44a7-c952-49b1-b5eb-6f527d149c02" (UID: "606c44a7-c952-49b1-b5eb-6f527d149c02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.025475 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.025432    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-workload-socket" pod "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" (UID: "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.025630 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.025606    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606c44a7-c952-49b1-b5eb-6f527d149c02-istiod-ca-cert" pod "606c44a7-c952-49b1-b5eb-6f527d149c02" (UID: "606c44a7-c952-49b1-b5eb-6f527d149c02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:45.025696 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.025633    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-workload-certs" pod "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" (UID: "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.026421 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.025859    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-credential-socket" pod "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" (UID: "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.026548 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.026480    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-workload-socket" pod "606c44a7-c952-49b1-b5eb-6f527d149c02" (UID: "606c44a7-c952-49b1-b5eb-6f527d149c02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.027125 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.027105    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.027125 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.027126    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.027362 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.027137    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.027362 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.027164    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.027362 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.027176    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.027362 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.027188    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606c44a7-c952-49b1-b5eb-6f527d149c02-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.027362 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.027204    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.027362 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.027220    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.027668 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.027511    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-data" pod "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" (UID: "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.028541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.028501    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istiod-ca-cert" pod "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" (UID: "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:45.028982 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.028957    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-envoy" pod "606c44a7-c952-49b1-b5eb-6f527d149c02" (UID: "606c44a7-c952-49b1-b5eb-6f527d149c02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.029077 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.029013    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-kube-api-access-f9q95" pod "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" (UID: "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757"). InnerVolumeSpecName "kube-api-access-f9q95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.029077 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.029066    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-podinfo" pod "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" (UID: "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:45.029237 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.029089    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606c44a7-c952-49b1-b5eb-6f527d149c02-kube-api-access-5fbtf" pod "606c44a7-c952-49b1-b5eb-6f527d149c02" (UID: "606c44a7-c952-49b1-b5eb-6f527d149c02"). InnerVolumeSpecName "kube-api-access-5fbtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.030559 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.030536    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-token" pod "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" (UID: "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.030759 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.030727    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-envoy" pod "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" (UID: "06905c96-e1c3-4ea0-b2c9-a47e7a9a3757"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.030874 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.030832    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-token" pod "606c44a7-c952-49b1-b5eb-6f527d149c02" (UID: "606c44a7-c952-49b1-b5eb-6f527d149c02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.030936 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.030913    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-podinfo" pod "606c44a7-c952-49b1-b5eb-6f527d149c02" (UID: "606c44a7-c952-49b1-b5eb-6f527d149c02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:45.129591 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.129457    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.129591 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.129494    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.129591 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.129508    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.129591 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.129521    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.129591 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.129534    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.129591 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.129547    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.129591 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.129559    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.129591 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.129574    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9q95\" (UniqueName: \"kubernetes.io/projected/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757-kube-api-access-f9q95\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.129591 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.129589    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606c44a7-c952-49b1-b5eb-6f527d149c02-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.129591 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.129606    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fbtf\" (UniqueName: \"kubernetes.io/projected/606c44a7-c952-49b1-b5eb-6f527d149c02-kube-api-access-5fbtf\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.534740 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.534704    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:45.556405 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.556381    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-dq4xb"
Oct 29 22:08:45.688965 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.688920    4550 generic.go:356] "Generic (PLEG): container finished" podID="06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" containerID="b862f9c98dbe483f54d78b0d306fff152e55854b8402cc84177057b361f195b7" exitCode=0
Oct 29 22:08:45.689200 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.689058    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55" event={"ID":"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757","Type":"ContainerDied","Data":"b862f9c98dbe483f54d78b0d306fff152e55854b8402cc84177057b361f195b7"}
Oct 29 22:08:45.689200 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.689115    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55" event={"ID":"06905c96-e1c3-4ea0-b2c9-a47e7a9a3757","Type":"ContainerDied","Data":"8db4358e39fd03022e7f2acb83658d69968a2f7dbfe305b0062ca24c1ec18e0c"}
Oct 29 22:08:45.689200 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.689162    4550 scope.go:122] "RemoveContainer" containerID="b862f9c98dbe483f54d78b0d306fff152e55854b8402cc84177057b361f195b7"
Oct 29 22:08:45.689381 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.689307    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"
Oct 29 22:08:45.691785 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.691762    4550 generic.go:356] "Generic (PLEG): container finished" podID="606c44a7-c952-49b1-b5eb-6f527d149c02" containerID="6d8d8a9ccd5ab2288670821dd9e99c207efbcee77811449a1bca29335d15aa73" exitCode=0
Oct 29 22:08:45.691914 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.691807    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz" event={"ID":"606c44a7-c952-49b1-b5eb-6f527d149c02","Type":"ContainerDied","Data":"6d8d8a9ccd5ab2288670821dd9e99c207efbcee77811449a1bca29335d15aa73"}
Oct 29 22:08:45.691914 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.691848    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz" event={"ID":"606c44a7-c952-49b1-b5eb-6f527d149c02","Type":"ContainerDied","Data":"e865d432a1ae083e59afc1c373a040253b8261ff88bed8a906bb92430f1f1d8b"}
Oct 29 22:08:45.691914 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.691809    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"
Oct 29 22:08:45.704800 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.704653    4550 scope.go:122] "RemoveContainer" containerID="b862f9c98dbe483f54d78b0d306fff152e55854b8402cc84177057b361f195b7"
Oct 29 22:08:45.705137 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:08:45.705112    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b862f9c98dbe483f54d78b0d306fff152e55854b8402cc84177057b361f195b7\": container with ID starting with b862f9c98dbe483f54d78b0d306fff152e55854b8402cc84177057b361f195b7 not found: ID does not exist" containerID="b862f9c98dbe483f54d78b0d306fff152e55854b8402cc84177057b361f195b7"
Oct 29 22:08:45.705261 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.705209    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b862f9c98dbe483f54d78b0d306fff152e55854b8402cc84177057b361f195b7"} err="failed to get container status \"b862f9c98dbe483f54d78b0d306fff152e55854b8402cc84177057b361f195b7\": rpc error: code = NotFound desc = could not find container \"b862f9c98dbe483f54d78b0d306fff152e55854b8402cc84177057b361f195b7\": container with ID starting with b862f9c98dbe483f54d78b0d306fff152e55854b8402cc84177057b361f195b7 not found: ID does not exist"
Oct 29 22:08:45.705261 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.705235    4550 scope.go:122] "RemoveContainer" containerID="6d8d8a9ccd5ab2288670821dd9e99c207efbcee77811449a1bca29335d15aa73"
Oct 29 22:08:45.726799 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.726765    4550 scope.go:122] "RemoveContainer" containerID="6d8d8a9ccd5ab2288670821dd9e99c207efbcee77811449a1bca29335d15aa73"
Oct 29 22:08:45.727309 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:08:45.727276    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d8d8a9ccd5ab2288670821dd9e99c207efbcee77811449a1bca29335d15aa73\": container with ID starting with 6d8d8a9ccd5ab2288670821dd9e99c207efbcee77811449a1bca29335d15aa73 not found: ID does not exist" containerID="6d8d8a9ccd5ab2288670821dd9e99c207efbcee77811449a1bca29335d15aa73"
Oct 29 22:08:45.727554 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.727317    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8d8a9ccd5ab2288670821dd9e99c207efbcee77811449a1bca29335d15aa73"} err="failed to get container status \"6d8d8a9ccd5ab2288670821dd9e99c207efbcee77811449a1bca29335d15aa73\": rpc error: code = NotFound desc = could not find container \"6d8d8a9ccd5ab2288670821dd9e99c207efbcee77811449a1bca29335d15aa73\": container with ID starting with 6d8d8a9ccd5ab2288670821dd9e99c207efbcee77811449a1bca29335d15aa73 not found: ID does not exist"
Oct 29 22:08:45.764244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.764211    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"]
Oct 29 22:08:45.768755 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.768731    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-74lvz"]
Oct 29 22:08:45.801523 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.801456    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"]
Oct 29 22:08:45.811386 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.811363    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7gq55"]
Oct 29 22:08:45.990992 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.990955    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" path="/var/lib/kubelet/pods/06905c96-e1c3-4ea0-b2c9-a47e7a9a3757/volumes"
Oct 29 22:08:45.991567 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:45.991548    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="606c44a7-c952-49b1-b5eb-6f527d149c02" path="/var/lib/kubelet/pods/606c44a7-c952-49b1-b5eb-6f527d149c02/volumes"
Oct 29 22:08:47.027837 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:47.027792    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:47.284995 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:47.284884    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:47.425759 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:47.425712    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:47.824093 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:47.824036    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:48.013896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:48.013861    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:48.276543 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:48.276444    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:48.414345 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:48.414291    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:48.580319 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:48.580283    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:50.286709 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:50.286661    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:50.720202 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:50.720166    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"]
Oct 29 22:08:50.831059 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:50.831021    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:50.960221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:50.956853    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:52.170680 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:52.170645    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:53.326441 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:53.326408    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:53.494168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:53.485567    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:54.517136 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:54.517103    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:55.720883 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:55.720832    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw" podUID="5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4" containerName="payload-pre-processing" containerID="cri-o://1562b28ea0328fe3cd4b46a22fc15ec5eb856c72a6ad5d4c41574c8dab88cdc7" gracePeriod=25
Oct 29 22:08:55.906392 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:55.906331    4550 generic.go:356] "Generic (PLEG): container finished" podID="5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4" containerID="1562b28ea0328fe3cd4b46a22fc15ec5eb856c72a6ad5d4c41574c8dab88cdc7" exitCode=0
Oct 29 22:08:55.906617 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:55.906465    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw" event={"ID":"5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4","Type":"ContainerDied","Data":"1562b28ea0328fe3cd4b46a22fc15ec5eb856c72a6ad5d4c41574c8dab88cdc7"}
Oct 29 22:08:56.026401 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:56.026370    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"
Oct 29 22:08:56.152475 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:56.152440    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4-plugins-config-volume\") pod \"5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4\" (UID: \"5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4\") "
Oct 29 22:08:56.152631 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:56.152614    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4-kube-api-access-t2q2r\" (UniqueName: \"kubernetes.io/projected/5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4-kube-api-access-t2q2r\") pod \"5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4\" (UID: \"5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4\") "
Oct 29 22:08:56.152898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:56.152869    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4-plugins-config-volume" pod "5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4" (UID: "5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:56.154174 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:56.153850    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4-plugins-config-volume\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:56.155869 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:56.155839    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4-kube-api-access-t2q2r" pod "5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4" (UID: "5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4"). InnerVolumeSpecName "kube-api-access-t2q2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:56.217868 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:56.217834    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:56.255855 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:56.255770    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2q2r\" (UniqueName: \"kubernetes.io/projected/5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4-kube-api-access-t2q2r\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:08:56.349295 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:56.349245    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:56.601867 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:56.601833    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:56.937828 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:56.936886    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw" event={"ID":"5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4","Type":"ContainerDied","Data":"af7add2acebad0f5f2cb645be99fa9b91dc1884db0702a1302827cb94b8473ec"}
Oct 29 22:08:56.937828 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:56.936969    4550 scope.go:122] "RemoveContainer" containerID="1562b28ea0328fe3cd4b46a22fc15ec5eb856c72a6ad5d4c41574c8dab88cdc7"
Oct 29 22:08:56.937828 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:56.937178    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"
Oct 29 22:08:56.998569 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:56.998534    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"]
Oct 29 22:08:57.006790 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:57.006751    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-301fcc-56cb7559b7-zpqlw"]
Oct 29 22:08:57.558782 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:57.558747    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:57.715096 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:57.715060    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:08:58.019517 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:58.019478    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4" path="/var/lib/kubelet/pods/5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4/volumes"
Oct 29 22:08:58.877381 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:08:58.877344    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:00.720000 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:00.719953    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:02.394094 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:02.394060    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:03.769000 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:03.768969    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:04.361675 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:04.361602    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:04.555900 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:04.555867    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:04.602131 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:04.602096    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"]
Oct 29 22:09:04.612651 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:04.612565    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a26f29-j9dpd"]
Oct 29 22:09:04.676546 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:04.676510    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:04.817175 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:04.811290    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:05.081197 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:05.081170    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:05.990719 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:05.990683    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c52c5564-6eef-4835-8f36-2f378dea76fb" path="/var/lib/kubelet/pods/c52c5564-6eef-4835-8f36-2f378dea76fb/volumes"
Oct 29 22:09:06.685963 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:06.685929    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:08.910764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:08.910724    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:10.540125 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:10.540094    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:11.492637 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:09:11.492604    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:09:11.877348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:11.877320    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:12.000948 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:12.000912    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:12.254037 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:12.253944    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:12.382098 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:12.381906    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:12.948218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:12.948129    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:13.231681 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:13.231610    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:13.735027 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:13.734990    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:13.884713 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:13.882599    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:14.477512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:14.474569    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:14.624298 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:14.617342    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:14.774236 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:14.774133    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:16.610387 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:16.610324    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:16.780615 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:16.780521    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:16.898348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:16.898258    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:17.135086 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:17.135029    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:17.557060 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:17.556755    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:17.845818 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:17.845768    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:17.968072 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:17.968036    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:18.139413 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:18.139304    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:18.969554 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:18.969517    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:19.098833 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:19.098788    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:19.313168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:19.313065    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:19.435605 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:19.435547    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:20.022034 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:20.019773    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:22.282163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:22.277058    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:22.337248 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:22.334320    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:22.608287 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:22.608238    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:09:22.624846 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:22.624792    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:22.651264 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:22.651221    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:09:22.768835 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:22.768245    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:09:22.855167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:22.848068    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:23.003332 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.003237    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:23.038163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.024234    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:23.194173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.193537    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:09:23.195968 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.195946    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:23.212467 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.212220    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:23.239726 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.237298    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:23.257311 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.257218    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:23.426218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.405525    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:09:23.426218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.406278    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:09:23.426218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.408262    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:09:23.731193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.727539    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:09:23.758171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.732009    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:09:23.760480 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.756997    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:23.895336 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.820817    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:09:23.896113 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.825637    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:23.897128 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.897109    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:09:23.920255 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.825825    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:09:24.205168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:24.199107    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:24.253173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:24.237324    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:09:24.282171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:24.279450    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:09:24.633291 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:24.037022    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:09:24.633291 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.835525    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:24.633291 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.895080    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:09:24.775352 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:24.775318    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:24.979635 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:24.979500    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:09:24.995310 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:23.826025    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:09:25.011812 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:24.942390    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.210943 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:24.948233    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.210943 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:24.949616    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.210943 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:24.915638    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.210943 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:25.207684    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.210943 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:25.210808    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.263958 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:25.237864    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.294227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:25.289204    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:09:25.458451 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:25.214628    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:09:25.654175 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:25.646506    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.980561 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:25.980458    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:09:26.229439 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.229397    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:26.238844 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.238727    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:26.498402 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.498253    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:26.539342 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.539294    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:09:26.572227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.572187    4550 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.358745974s: [/var/lib/containers/storage/overlay/b478c8a2f220d829d7164280dd6beebc36f5bfbd9d5cee7a4226e83204b31cab/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:09:26.666037 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.507270    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:09:26.687472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.687426    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" podUID="93b4187a-1551-4cf8-b807-d057baf03920" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:26.687824 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.687770    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:09:26.825219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.810215    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:09:26.825219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.810806    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:09:26.825219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.814556    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:09:26.825219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.818629    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:26.932315 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.932269    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:26.933125 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.933095    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:26.933736 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.933712    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:26.933886 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:26.933865    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:27.049489 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.047442    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:27.049489 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:09:27.048032    4550 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.55s"
Oct 29 22:09:27.091254 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.083239    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:27.176165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.168321    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:27.238319 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.238278    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:27.273079 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.273045    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:09:27.695304 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.695255    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:09:27.703276 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.703232    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:09:27.930464 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.714212    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:27.951274 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.951222    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 29 22:09:27.951543 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.897670    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:09:27.964839 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.729422    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:09:27.966122 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.897709    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:09:27.973384 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.897741    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:09:28.026718 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.929502    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:09:28.027568 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.966824    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:09:28.027802 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.027782    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:09:28.084201 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:27.897761    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:09:28.084201 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.052287    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.151289 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.151248    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.246300 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.176669    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:09:28.259246 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.256245    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:09:28.353179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.341194    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:28.451265 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.451209    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:28.455164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.451512    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp" podUID="7fe0d723-a137-4c93-9657-ca9258b08a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.568709 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.568669    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:28.696212 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.696169    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.845391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.845354    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:09:28.845655 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.845636    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:09:28.845820 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.845804    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"]
Oct 29 22:09:28.845998 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.845983    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:09:28.846204 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.846188    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:09:28.846350 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.846338    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:09:28.896722 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.853228    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w" podUID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerName="istio-proxy" containerID="cri-o://96088a4b69e2f13ec068d23436a627ea1ae04a48f43d8ee362fe4cf742f200bf" gracePeriod=30
Oct 29 22:09:28.939117 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.923433    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.939117 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.929741    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.939117 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.930272    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.979175 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.968706    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.979756 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:28.979715    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:09:29.059166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:29.049055    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.170978 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:29.111762    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:09:29.170978 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:29.112042    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.170978 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:29.116902    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.261171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:29.240885    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.261171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:29.259228    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:09:29.266170 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:29.265975    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:09:29.379020 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:29.378984    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:29.743167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:29.742883    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:09:29.885169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:29.825335    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:29.885169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:29.825413    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.952165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:29.944557    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.952165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:29.945220    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:09:29.978230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:29.978175    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:09:30.050368 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:30.050266    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.051374 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:30.051335    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.122846 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:30.122800    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.124583 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:30.124528    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:09:30.126248 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:30.126220    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.250122 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:30.250077    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.280980 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:30.275373    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:09:30.310176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:30.306885    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.516165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:30.515208    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.619167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:30.610579    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.690168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:30.680405    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded"
Oct 29 22:09:30.745607 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:30.743326    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.034255 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.007976    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:09:31.034255 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.008393    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.051911 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.051866    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.052630 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.052597    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:09:31.172737 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.172685    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.175666 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.175611    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:09:31.176258 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.176231    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:09:31.195733 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.195690    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:09:31.233166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.229427    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:31.281170 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.259517    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:09:31.315778 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.281482    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.316683 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.307968    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.387635 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.315281    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" podUID="93b4187a-1551-4cf8-b807-d057baf03920" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.528827 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.482685    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" podUID="93b4187a-1551-4cf8-b807-d057baf03920" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.581227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.581157    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.757491 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:31.757451    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:09:32.015376 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:32.015255    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.037469 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:32.037425    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:09:32.053719 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:32.053667    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.054448 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:32.054412    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:09:32.186447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:32.176729    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.186447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:32.177131    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.186447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:32.177457    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:09:32.234335 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:32.233305    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.249213 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:32.248646    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:32.249213 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:32.248717    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.351229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:32.338796    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.351229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:32.339197    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:09:32.403340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:32.388375    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.699222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:32.698850    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.764794 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:32.764749    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.029236 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:33.017206    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.047027 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:33.046290    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.070215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:33.061492    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.070215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:33.061886    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:09:33.103181 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:33.088345    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.184831 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:33.184601    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.188709 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:33.184697    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:09:33.219798 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:33.219457    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.238946 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:33.238319    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.246170 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:33.239252    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.346392 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:33.344239    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:09:33.346392 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:33.345115    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:09:33.393423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:33.389576    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.768087 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:33.765759    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.862464 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:33.862212    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 29 22:09:34.066328 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.066208    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.066765 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.066738    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:09:34.109904 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.067240    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.110414 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.092243    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.191597 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.191546    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.192379 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.192347    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:09:34.252176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.249069    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.252176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.249527    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:09:34.351166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.347288    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.351166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.347757    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:09:34.402166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.398824    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.607161 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.603319    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:34.607161 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.603382    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7" podUID="a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.650172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.649236    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:34.650172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.649302    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.717714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.708299    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:34.717714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.708368    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" podUID="a26797f1-f34c-44e3-86e9-dd0ec24af003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.767192 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.766843    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:09:34.932884 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:34.883316    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:09:35.173763 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.124317    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:09:35.173763 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.124709    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:09:35.173763 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.125269    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.247718 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.213852    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.257024 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.250319    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.257024 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.250678    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:09:35.267615 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.267250    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.315540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.315221    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:09:35.572219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.518312    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.572219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.519205    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:09:35.572219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.520253    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:09:35.572219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.547972    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-envoy\") pod \"097cc026-11a8-45b2-8eda-26fce58e7a02\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") "
Oct 29 22:09:35.572219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.548096    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-token\" (UniqueName: \"kubernetes.io/projected/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-token\") pod \"097cc026-11a8-45b2-8eda-26fce58e7a02\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") "
Oct 29 22:09:35.572219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.548178    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-workload-socket\") pod \"097cc026-11a8-45b2-8eda-26fce58e7a02\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") "
Oct 29 22:09:35.572219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.548221    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-workload-certs\") pod \"097cc026-11a8-45b2-8eda-26fce58e7a02\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") "
Oct 29 22:09:35.572219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.548378    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-data\") pod \"097cc026-11a8-45b2-8eda-26fce58e7a02\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") "
Oct 29 22:09:35.572219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.548461    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-podinfo\") pod \"097cc026-11a8-45b2-8eda-26fce58e7a02\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") "
Oct 29 22:09:35.572219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.548591    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/097cc026-11a8-45b2-8eda-26fce58e7a02-kube-api-access-h9bxn\" (UniqueName: \"kubernetes.io/projected/097cc026-11a8-45b2-8eda-26fce58e7a02-kube-api-access-h9bxn\") pod \"097cc026-11a8-45b2-8eda-26fce58e7a02\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") "
Oct 29 22:09:35.572219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.548665    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/097cc026-11a8-45b2-8eda-26fce58e7a02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/097cc026-11a8-45b2-8eda-26fce58e7a02-istiod-ca-cert\") pod \"097cc026-11a8-45b2-8eda-26fce58e7a02\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") "
Oct 29 22:09:35.572219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.549931    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-workload-socket" pod "097cc026-11a8-45b2-8eda-26fce58e7a02" (UID: "097cc026-11a8-45b2-8eda-26fce58e7a02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:09:35.572955 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.572669    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-workload-certs" pod "097cc026-11a8-45b2-8eda-26fce58e7a02" (UID: "097cc026-11a8-45b2-8eda-26fce58e7a02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:09:35.586176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.573460    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:35.586176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.573513    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.586176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.574085    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097cc026-11a8-45b2-8eda-26fce58e7a02-istiod-ca-cert" pod "097cc026-11a8-45b2-8eda-26fce58e7a02" (UID: "097cc026-11a8-45b2-8eda-26fce58e7a02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:09:35.639349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.595256    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-data" pod "097cc026-11a8-45b2-8eda-26fce58e7a02" (UID: "097cc026-11a8-45b2-8eda-26fce58e7a02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:09:35.681244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.674749    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-credential-socket" pod "097cc026-11a8-45b2-8eda-26fce58e7a02" (UID: "097cc026-11a8-45b2-8eda-26fce58e7a02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:09:35.681244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.674803    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-credential-socket\") pod \"097cc026-11a8-45b2-8eda-26fce58e7a02\" (UID: \"097cc026-11a8-45b2-8eda-26fce58e7a02\") "
Oct 29 22:09:35.681244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.676521    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/097cc026-11a8-45b2-8eda-26fce58e7a02-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:09:35.681244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.676573    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:09:35.681244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.676588    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:09:35.681244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.676623    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:09:35.681244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.676641    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:09:35.778249 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.776660    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.885854 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.882781    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-token" pod "097cc026-11a8-45b2-8eda-26fce58e7a02" (UID: "097cc026-11a8-45b2-8eda-26fce58e7a02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:35.885854 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.882899    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-envoy" pod "097cc026-11a8-45b2-8eda-26fce58e7a02" (UID: "097cc026-11a8-45b2-8eda-26fce58e7a02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:09:35.885854 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.885189    4550 generic.go:356] "Generic (PLEG): container finished" podID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerID="96088a4b69e2f13ec068d23436a627ea1ae04a48f43d8ee362fe4cf742f200bf" exitCode=0
Oct 29 22:09:35.885854 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.885448    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w" event={"ID":"097cc026-11a8-45b2-8eda-26fce58e7a02","Type":"ContainerDied","Data":"96088a4b69e2f13ec068d23436a627ea1ae04a48f43d8ee362fe4cf742f200bf"}
Oct 29 22:09:35.885854 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.885507    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w" event={"ID":"097cc026-11a8-45b2-8eda-26fce58e7a02","Type":"ContainerDied","Data":"d00a5269cea9969865f1c90a6a25f8bbe60b0bce05f8b97b9a19819b5a12ffb1"}
Oct 29 22:09:35.885854 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.885544    4550 scope.go:122] "RemoveContainer" containerID="96088a4b69e2f13ec068d23436a627ea1ae04a48f43d8ee362fe4cf742f200bf"
Oct 29 22:09:35.885854 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.885809    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-podinfo" pod "097cc026-11a8-45b2-8eda-26fce58e7a02" (UID: "097cc026-11a8-45b2-8eda-26fce58e7a02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:09:35.914423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.908508    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"
Oct 29 22:09:35.970908 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.956063    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:09:35.970908 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.956098    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:09:35.970908 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.956112    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/097cc026-11a8-45b2-8eda-26fce58e7a02-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:09:35.970908 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.959681    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.970908 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:35.970300    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097cc026-11a8-45b2-8eda-26fce58e7a02-kube-api-access-h9bxn" pod "097cc026-11a8-45b2-8eda-26fce58e7a02" (UID: "097cc026-11a8-45b2-8eda-26fce58e7a02"). InnerVolumeSpecName "kube-api-access-h9bxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:36.039656 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.030780    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:36.110787 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.103392    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9bxn\" (UniqueName: \"kubernetes.io/projected/097cc026-11a8-45b2-8eda-26fce58e7a02-kube-api-access-h9bxn\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:09:36.153236 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.132540    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.153236 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.133362    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:09:36.153236 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.135133    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:09:36.220167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.215478    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.275623 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.250611    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.275623 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.252702    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:09:36.275623 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.274778    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.324053 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.323728    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.333530 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.333410    4550 scope.go:122] "RemoveContainer" containerID="a2746305a38ab0891c511dd8ef7e79ff52f59c93741d58dc873b85ca742988bb"
Oct 29 22:09:36.486250 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.483224    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"]
Oct 29 22:09:36.507197 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.502476    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d53cd-openshift-default-77fdb9d4c-6b62w"]
Oct 29 22:09:36.545298 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.541390    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.545298 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.542707    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:09:36.560898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.560531    4550 scope.go:122] "RemoveContainer" containerID="96088a4b69e2f13ec068d23436a627ea1ae04a48f43d8ee362fe4cf742f200bf"
Oct 29 22:09:36.565614 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:09:36.565175    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96088a4b69e2f13ec068d23436a627ea1ae04a48f43d8ee362fe4cf742f200bf\": container with ID starting with 96088a4b69e2f13ec068d23436a627ea1ae04a48f43d8ee362fe4cf742f200bf not found: ID does not exist" containerID="96088a4b69e2f13ec068d23436a627ea1ae04a48f43d8ee362fe4cf742f200bf"
Oct 29 22:09:36.565614 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.565236    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96088a4b69e2f13ec068d23436a627ea1ae04a48f43d8ee362fe4cf742f200bf"} err="failed to get container status \"96088a4b69e2f13ec068d23436a627ea1ae04a48f43d8ee362fe4cf742f200bf\": rpc error: code = NotFound desc = could not find container \"96088a4b69e2f13ec068d23436a627ea1ae04a48f43d8ee362fe4cf742f200bf\": container with ID starting with 96088a4b69e2f13ec068d23436a627ea1ae04a48f43d8ee362fe4cf742f200bf not found: ID does not exist"
Oct 29 22:09:36.565614 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.565269    4550 scope.go:122] "RemoveContainer" containerID="a2746305a38ab0891c511dd8ef7e79ff52f59c93741d58dc873b85ca742988bb"
Oct 29 22:09:36.567547 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:09:36.566026    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2746305a38ab0891c511dd8ef7e79ff52f59c93741d58dc873b85ca742988bb\": container with ID starting with a2746305a38ab0891c511dd8ef7e79ff52f59c93741d58dc873b85ca742988bb not found: ID does not exist" containerID="a2746305a38ab0891c511dd8ef7e79ff52f59c93741d58dc873b85ca742988bb"
Oct 29 22:09:36.567547 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.566061    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2746305a38ab0891c511dd8ef7e79ff52f59c93741d58dc873b85ca742988bb"} err="failed to get container status \"a2746305a38ab0891c511dd8ef7e79ff52f59c93741d58dc873b85ca742988bb\": rpc error: code = NotFound desc = could not find container \"a2746305a38ab0891c511dd8ef7e79ff52f59c93741d58dc873b85ca742988bb\": container with ID starting with a2746305a38ab0891c511dd8ef7e79ff52f59c93741d58dc873b85ca742988bb not found: ID does not exist"
Oct 29 22:09:36.712421 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.702796    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:09:36.778576 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.778071    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.878571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.878542    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:09:36.915087 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.914984    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:09:36.977184 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.974253    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:09:36.977184 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:36.974873    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:09:37.060444 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:37.060331    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:09:37.061985 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:37.061962    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:09:37.097185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:37.090649    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:09:37.145322 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:37.140011    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:09:37.145322 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:37.140556    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:37.150372 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:37.150342    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:09:37.177665 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:37.177641    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:09:37.190488 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:37.190455    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:09:37.218584 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:37.216934    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:37.223347 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:37.223319    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:09:37.304131 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:37.304096    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:09:37.304555 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:37.304526    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:09:37.304764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:37.304750    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:09:38.002998 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:38.002966    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="097cc026-11a8-45b2-8eda-26fce58e7a02" path="/var/lib/kubelet/pods/097cc026-11a8-45b2-8eda-26fce58e7a02/volumes"
Oct 29 22:09:38.290586 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:38.290440    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:40.661582 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:40.661534    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:41.492468 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:09:41.492426    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:09:42.416274 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:42.416240    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:43.898341 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:43.898306    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:44.028907 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:44.028865    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:44.371068 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:44.371020    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:45.108906 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:45.108871    4550 generic.go:356] "Generic (PLEG): container finished" podID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerID="1b27f106dfc515e505cf13888d762b8646cb0fb0d18481889c1465b92b5398c0" exitCode=0
Oct 29 22:09:45.109407 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:45.108959    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" event={"ID":"aec85ac8-9f07-402f-98d3-cb8b5f5fe581","Type":"ContainerDied","Data":"1b27f106dfc515e505cf13888d762b8646cb0fb0d18481889c1465b92b5398c0"}
Oct 29 22:09:45.109452 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:45.109409    4550 scope.go:122] "RemoveContainer" containerID="1b27f106dfc515e505cf13888d762b8646cb0fb0d18481889c1465b92b5398c0"
Oct 29 22:09:45.111404 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:45.111369    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:09:45.111549 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:45.111449    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:09:45.111549 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:45.111491    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:09:45.565689 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:45.565594    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:45.744995 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:45.744943    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:46.122892 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:46.122851    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" event={"ID":"aec85ac8-9f07-402f-98d3-cb8b5f5fe581","Type":"ContainerStarted","Data":"b3c6b3db187f2c39510c7a0d8ed423d7026c018c09abae1bc3c267cadf1372b2"}
Oct 29 22:09:46.281168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:46.277991    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:46.849631 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:46.849598    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:09:46.849854 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:46.849792    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:09:46.849854 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:46.849843    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:09:47.820860 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.820818    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"]
Oct 29 22:09:47.821859 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.821743    4550 cpu_manager.go:414] "containerMap: removing container" podUID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerName="istio-proxy"
Oct 29 22:09:47.822021 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.822005    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerName="istio-proxy"
Oct 29 22:09:47.822117 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.822106    4550 cpu_manager.go:414] "containerMap: removing container" podUID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerName="istio-proxy"
Oct 29 22:09:47.822221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.822210    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerName="istio-proxy"
Oct 29 22:09:47.822310 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.822301    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4" containerName="payload-pre-processing"
Oct 29 22:09:47.822378 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.822368    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4" containerName="payload-pre-processing"
Oct 29 22:09:47.822501 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.822491    4550 cpu_manager.go:414] "containerMap: removing container" podUID="06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" containerName="istio-proxy"
Oct 29 22:09:47.822574 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.822564    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" containerName="istio-proxy"
Oct 29 22:09:47.822663 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.822652    4550 cpu_manager.go:414] "containerMap: removing container" podUID="606c44a7-c952-49b1-b5eb-6f527d149c02" containerName="istio-proxy"
Oct 29 22:09:47.822738 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.822729    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606c44a7-c952-49b1-b5eb-6f527d149c02" containerName="istio-proxy"
Oct 29 22:09:47.823007 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.822992    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="06905c96-e1c3-4ea0-b2c9-a47e7a9a3757" containerName="istio-proxy"
Oct 29 22:09:47.823123 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.823112    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="606c44a7-c952-49b1-b5eb-6f527d149c02" containerName="istio-proxy"
Oct 29 22:09:47.823235 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.823226    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerName="istio-proxy"
Oct 29 22:09:47.823294 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.823287    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="097cc026-11a8-45b2-8eda-26fce58e7a02" containerName="istio-proxy"
Oct 29 22:09:47.823348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.823342    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a2f3c23-25b5-4a2d-b44b-765cfb4e51d4" containerName="payload-pre-processing"
Oct 29 22:09:47.853511 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.853476    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:09:47.853796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.853751    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:09:47.854166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.854127    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"]
Oct 29 22:09:47.854545 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.854311    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:47.877214 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.871448    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-df42c7-openshift-default-dockercfg-jqslj\""
Oct 29 22:09:47.925040 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.924995    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:47.934980 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.934938    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:47.935188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.935007    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:47.935188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.935067    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:47.935188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.935096    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:47.935761 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.935731    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:47.935842 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.935778    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:47.935842 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.935810    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtzmt\" (UniqueName: \"kubernetes.io/projected/1be95f05-57d1-4e36-8fd4-96b90281dcee-kube-api-access-rtzmt\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:47.936004 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.935863    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be95f05-57d1-4e36-8fd4-96b90281dcee-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:47.936004 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:47.935916    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.040153 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.040103    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.040347 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.040180    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.040347 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.040246    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.040347 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.040276    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.040795 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.040770    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.041459 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.041437    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.041575 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.041558    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.041639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.041607    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.041694 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.041637    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtzmt\" (UniqueName: \"kubernetes.io/projected/1be95f05-57d1-4e36-8fd4-96b90281dcee-kube-api-access-rtzmt\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.041694 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.041687    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be95f05-57d1-4e36-8fd4-96b90281dcee-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.041806 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.041742    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.042756 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.042726    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.042973 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.042952    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.044051 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.044022    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be95f05-57d1-4e36-8fd4-96b90281dcee-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.051041 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.045278    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.051041 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.049684    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.122941 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.122883    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:48.151636 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.151320    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtzmt\" (UniqueName: \"kubernetes.io/projected/1be95f05-57d1-4e36-8fd4-96b90281dcee-kube-api-access-rtzmt\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.157121 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.153447    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.233539 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.221424    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:48.255633 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.241474    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:48.569167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.567956    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:48.737242 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.720129    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:48.885196 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.885078    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:09:48.886063 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:48.885682    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:09:49.110494 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:49.110466    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"]
Oct 29 22:09:49.111618 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:49.111589    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:49.112941 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:49.112907    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:09:49.113055 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:49.112985    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:09:49.113055 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:49.113015    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:09:49.152570 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:49.152546    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d" event={"ID":"1be95f05-57d1-4e36-8fd4-96b90281dcee","Type":"ContainerStarted","Data":"c6276be1b3f8f9f3125a2361ef4296a48744eb49f554e3c7cb42f10986001e95"}
Oct 29 22:09:49.229392 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:49.229362    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:49.756502 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:49.756468    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:49.850527 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:49.850494    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:09:49.850704 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:49.850548    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:09:49.991978 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:49.991948    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:50.186168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:50.185175    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d" event={"ID":"1be95f05-57d1-4e36-8fd4-96b90281dcee","Type":"ContainerStarted","Data":"c263da465c0f5c507cc7b14a790d1d8da15be1fddd87c6d1ecbda609b67a557c"}
Oct 29 22:09:50.226187 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:50.226129    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:50.227400 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:50.227374    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:09:50.227525 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:50.227419    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d" podUID="1be95f05-57d1-4e36-8fd4-96b90281dcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:09:50.245649 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:50.245607    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d" podStartSLOduration=3.245595428 podStartE2EDuration="3.245595428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:50.244964083 +0000 UTC m=+1406.402089606" watchObservedRunningTime="2026-10-29 22:09:50.245595428 +0000 UTC m=+1406.402720926"
Oct 29 22:09:50.498966 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:50.498887    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:50.628803 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:50.628768    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:50.872167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:50.871581    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:09:50.872167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:50.871640    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:09:51.228352 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:51.225629    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:09:51.228352 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:51.225692    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d" podUID="1be95f05-57d1-4e36-8fd4-96b90281dcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:09:51.546660 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:51.545767    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:51.697552 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:51.695991    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:51.840163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:51.832337    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:51.849579 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:51.849547    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:09:51.849842 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:51.849814    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:09:52.238221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:52.233587    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:09:52.238221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:52.233649    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d" podUID="1be95f05-57d1-4e36-8fd4-96b90281dcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:09:52.853994 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:52.853964    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:09:52.855127 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:52.855101    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:09:53.225742 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:53.225664    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:09:53.225742 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:53.225713    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d" podUID="1be95f05-57d1-4e36-8fd4-96b90281dcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:09:53.344213 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:53.344180    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:09:53.629801 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:53.629765    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:54.225841 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:54.225805    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:09:54.226021 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:54.225876    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d" podUID="1be95f05-57d1-4e36-8fd4-96b90281dcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:09:55.225649 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:55.225607    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:09:55.226168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:55.225702    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d" podUID="1be95f05-57d1-4e36-8fd4-96b90281dcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:09:55.352455 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:55.352420    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:56.226021 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:56.225992    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:09:56.226437 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:56.226054    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d" podUID="1be95f05-57d1-4e36-8fd4-96b90281dcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:09:56.797540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:56.797501    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:56.922202 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:56.922163    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:57.249356 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:57.249308    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"]
Oct 29 22:09:57.341179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:57.312447    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:57.415905 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:57.415871    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"]
Oct 29 22:09:57.416258 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:57.416225    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8" podUID="7c72b9b9-b934-4279-960c-d69fadadb664" containerName="istio-proxy" containerID="cri-o://28f101b46402fa9ee6400a5e40a8254b1cacfe618b54879cee5f349aa9c06a57" gracePeriod=30
Oct 29 22:09:57.417400 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:57.416245    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:57.417400 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:57.416837    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"]
Oct 29 22:09:57.417400 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:57.416952    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg" podUID="b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" containerName="istio-proxy" containerID="cri-o://6f3ee1dcbbb8e825cdc92cfb70a9856c368824bf45f96399e1181b71728a6058" gracePeriod=30
Oct 29 22:09:57.417400 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:57.417022    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq" podUID="a07ed650-5c07-4440-b64d-cb65a9529625" containerName="istio-proxy" containerID="cri-o://9b537ee11bf475283e067cdbaa66a2cac75bf40ec171a77e6ef3e1d4d8b0b437" gracePeriod=30
Oct 29 22:09:57.429445 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:57.429423    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:57.430496 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:57.430480    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:09:57.602676 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:57.602641    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:58.320077 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:58.320041    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:58.549503 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:58.549469    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:58.677768 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:58.677710    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:58.826535 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:58.826486    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:09:59.088613 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:09:59.088582    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:00.512566 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:00.512495    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:02.073219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.073078    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:10:02.073749 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.073260    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg" podUID="b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:10:02.283364 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.283322    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:02.409615 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.409580    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:02.534898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.534864    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:02.795743 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.795711    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.888865    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istiod-ca-cert\") pod \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") "
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.888946    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-workload-certs\") pod \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") "
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.889039    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-token\" (UniqueName: \"kubernetes.io/projected/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-token\") pod \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") "
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.889117    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-podinfo\") pod \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") "
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.889170    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-envoy\") pod \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") "
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.889224    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-credential-socket\") pod \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") "
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.889341    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-data\") pod \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") "
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.889445    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-workload-socket\") pod \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") "
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.889488    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-kube-api-access-tjcj2\" (UniqueName: \"kubernetes.io/projected/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-kube-api-access-tjcj2\") pod \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\" (UID: \"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7\") "
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.890863    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-credential-socket" pod "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" (UID: "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.891232    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-data" pod "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" (UID: "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.891423    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-workload-socket" pod "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" (UID: "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.892360    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istiod-ca-cert" pod "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" (UID: "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.892581    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-workload-certs" pod "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" (UID: "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.894470    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-podinfo" pod "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" (UID: "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.896778    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-envoy" pod "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" (UID: "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:02.897670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.897135    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-token" pod "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" (UID: "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:02.898761 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.897836    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-kube-api-access-tjcj2" pod "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" (UID: "b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7"). InnerVolumeSpecName "kube-api-access-tjcj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:02.931906 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.931880    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:10:02.934739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.934712    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:10:02.978741 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.978585    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"]
Oct 29 22:10:02.981644 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.981615    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" containerName="istio-proxy"
Oct 29 22:10:02.981863 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.981848    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" containerName="istio-proxy"
Oct 29 22:10:02.990717 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.990690    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7c72b9b9-b934-4279-960c-d69fadadb664" containerName="istio-proxy"
Oct 29 22:10:02.990901 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.990883    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c72b9b9-b934-4279-960c-d69fadadb664" containerName="istio-proxy"
Oct 29 22:10:02.991039 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.991026    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a07ed650-5c07-4440-b64d-cb65a9529625" containerName="istio-proxy"
Oct 29 22:10:02.991157 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.991127    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07ed650-5c07-4440-b64d-cb65a9529625" containerName="istio-proxy"
Oct 29 22:10:02.992925 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.992763    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:02.992925 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.992794    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:02.992925 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.992809    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:02.992925 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.992824    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:02.992925 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.992839    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:02.992925 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.992853    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:02.992925 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.992867    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjcj2\" (UniqueName: \"kubernetes.io/projected/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-kube-api-access-tjcj2\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:02.992925 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.992882    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:02.992925 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.992896    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:02.993722 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.993698    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c72b9b9-b934-4279-960c-d69fadadb664" containerName="istio-proxy"
Oct 29 22:10:02.993852 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.993839    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07ed650-5c07-4440-b64d-cb65a9529625" containerName="istio-proxy"
Oct 29 22:10:02.993961 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:02.993948    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" containerName="istio-proxy"
Oct 29 22:10:03.006790 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.005641    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.049920 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.049891    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"]
Oct 29 22:10:03.101389 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.101206    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-istio-envoy\") pod \"7c72b9b9-b934-4279-960c-d69fadadb664\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") "
Oct 29 22:10:03.101928 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.101439    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07ed650-5c07-4440-b64d-cb65a9529625-kube-api-access-xsrk8\" (UniqueName: \"kubernetes.io/projected/a07ed650-5c07-4440-b64d-cb65a9529625-kube-api-access-xsrk8\") pod \"a07ed650-5c07-4440-b64d-cb65a9529625\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") "
Oct 29 22:10:03.102283 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.102221    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-istio-data\") pod \"7c72b9b9-b934-4279-960c-d69fadadb664\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") "
Oct 29 22:10:03.102815 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.102678    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-istio-data" pod "7c72b9b9-b934-4279-960c-d69fadadb664" (UID: "7c72b9b9-b934-4279-960c-d69fadadb664"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.102962 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.102759    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-credential-socket\") pod \"a07ed650-5c07-4440-b64d-cb65a9529625\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") "
Oct 29 22:10:03.103136 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.103116    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-workload-certs\") pod \"a07ed650-5c07-4440-b64d-cb65a9529625\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") "
Oct 29 22:10:03.103304 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.103286    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c72b9b9-b934-4279-960c-d69fadadb664-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c72b9b9-b934-4279-960c-d69fadadb664-istiod-ca-cert\") pod \"7c72b9b9-b934-4279-960c-d69fadadb664\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") "
Oct 29 22:10:03.103457 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.103441    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07ed650-5c07-4440-b64d-cb65a9529625-istio-token\" (UniqueName: \"kubernetes.io/projected/a07ed650-5c07-4440-b64d-cb65a9529625-istio-token\") pod \"a07ed650-5c07-4440-b64d-cb65a9529625\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") "
Oct 29 22:10:03.103691 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.103668    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c72b9b9-b934-4279-960c-d69fadadb664-kube-api-access-wnvft\" (UniqueName: \"kubernetes.io/projected/7c72b9b9-b934-4279-960c-d69fadadb664-kube-api-access-wnvft\") pod \"7c72b9b9-b934-4279-960c-d69fadadb664\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") "
Oct 29 22:10:03.103883 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.103865    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-workload-socket\") pod \"7c72b9b9-b934-4279-960c-d69fadadb664\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") "
Oct 29 22:10:03.104063 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.104046    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-workload-socket\") pod \"a07ed650-5c07-4440-b64d-cb65a9529625\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") "
Oct 29 22:10:03.104276 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.104239    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c72b9b9-b934-4279-960c-d69fadadb664-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c72b9b9-b934-4279-960c-d69fadadb664-istio-podinfo\") pod \"7c72b9b9-b934-4279-960c-d69fadadb664\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") "
Oct 29 22:10:03.104901 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.103196    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-credential-socket" pod "a07ed650-5c07-4440-b64d-cb65a9529625" (UID: "a07ed650-5c07-4440-b64d-cb65a9529625"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.105003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.103437    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-workload-certs" pod "a07ed650-5c07-4440-b64d-cb65a9529625" (UID: "a07ed650-5c07-4440-b64d-cb65a9529625"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.105003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.103761    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c72b9b9-b934-4279-960c-d69fadadb664-istiod-ca-cert" pod "7c72b9b9-b934-4279-960c-d69fadadb664" (UID: "7c72b9b9-b934-4279-960c-d69fadadb664"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:03.105003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.104531    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-workload-socket" pod "a07ed650-5c07-4440-b64d-cb65a9529625" (UID: "a07ed650-5c07-4440-b64d-cb65a9529625"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.105003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.104811    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-workload-socket" pod "7c72b9b9-b934-4279-960c-d69fadadb664" (UID: "7c72b9b9-b934-4279-960c-d69fadadb664"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.105706 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.105690    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c72b9b9-b934-4279-960c-d69fadadb664-istio-token\" (UniqueName: \"kubernetes.io/projected/7c72b9b9-b934-4279-960c-d69fadadb664-istio-token\") pod \"7c72b9b9-b934-4279-960c-d69fadadb664\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") "
Oct 29 22:10:03.106047 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.106032    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-credential-socket\") pod \"7c72b9b9-b934-4279-960c-d69fadadb664\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") "
Oct 29 22:10:03.106265 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.106248    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-istio-data\") pod \"a07ed650-5c07-4440-b64d-cb65a9529625\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") "
Oct 29 22:10:03.106508 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.106492    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07ed650-5c07-4440-b64d-cb65a9529625-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07ed650-5c07-4440-b64d-cb65a9529625-istiod-ca-cert\") pod \"a07ed650-5c07-4440-b64d-cb65a9529625\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") "
Oct 29 22:10:03.106737 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.106720    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-workload-certs\") pod \"7c72b9b9-b934-4279-960c-d69fadadb664\" (UID: \"7c72b9b9-b934-4279-960c-d69fadadb664\") "
Oct 29 22:10:03.106867 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.106852    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a07ed650-5c07-4440-b64d-cb65a9529625-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07ed650-5c07-4440-b64d-cb65a9529625-istio-podinfo\") pod \"a07ed650-5c07-4440-b64d-cb65a9529625\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") "
Oct 29 22:10:03.106983 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.106969    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-istio-envoy\") pod \"a07ed650-5c07-4440-b64d-cb65a9529625\" (UID: \"a07ed650-5c07-4440-b64d-cb65a9529625\") "
Oct 29 22:10:03.107354 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.107330    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.107486 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.106337    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-credential-socket" pod "7c72b9b9-b934-4279-960c-d69fadadb664" (UID: "7c72b9b9-b934-4279-960c-d69fadadb664"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.107538 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.106499    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-istio-data" pod "a07ed650-5c07-4440-b64d-cb65a9529625" (UID: "a07ed650-5c07-4440-b64d-cb65a9529625"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.107538 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.106764    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07ed650-5c07-4440-b64d-cb65a9529625-istiod-ca-cert" pod "a07ed650-5c07-4440-b64d-cb65a9529625" (UID: "a07ed650-5c07-4440-b64d-cb65a9529625"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:03.107538 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.106993    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-workload-certs" pod "7c72b9b9-b934-4279-960c-d69fadadb664" (UID: "7c72b9b9-b934-4279-960c-d69fadadb664"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.107538 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.107447    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c72b9b9-b934-4279-960c-d69fadadb664-kube-api-access-wnvft" pod "7c72b9b9-b934-4279-960c-d69fadadb664" (UID: "7c72b9b9-b934-4279-960c-d69fadadb664"). InnerVolumeSpecName "kube-api-access-wnvft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:03.107691 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.107678    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.107834 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.107822    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.107975 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.107963    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.108053 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.108043    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.108204 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.108189    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.108320 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.108310    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.108454 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.108443    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgp4f\" (UniqueName: \"kubernetes.io/projected/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-kube-api-access-hgp4f\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.108519 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.108511    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.108734 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.108723    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.108821 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.108813    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.108883 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.108873    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.108938 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.108931    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.108996 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.108989    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c72b9b9-b934-4279-960c-d69fadadb664-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.109049 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.109041    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnvft\" (UniqueName: \"kubernetes.io/projected/7c72b9b9-b934-4279-960c-d69fadadb664-kube-api-access-wnvft\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.109103 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.109095    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.109188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.109174    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.109283 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.109270    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.109373 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.109361    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.109463 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.109452    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07ed650-5c07-4440-b64d-cb65a9529625-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.109542 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.109452    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a07ed650-5c07-4440-b64d-cb65a9529625-istio-podinfo" pod "a07ed650-5c07-4440-b64d-cb65a9529625" (UID: "a07ed650-5c07-4440-b64d-cb65a9529625"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:03.110027 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.109955    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-istio-envoy" pod "a07ed650-5c07-4440-b64d-cb65a9529625" (UID: "a07ed650-5c07-4440-b64d-cb65a9529625"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.110027 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.109998    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c72b9b9-b934-4279-960c-d69fadadb664-istio-token" pod "7c72b9b9-b934-4279-960c-d69fadadb664" (UID: "7c72b9b9-b934-4279-960c-d69fadadb664"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:03.111593 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.111560    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07ed650-5c07-4440-b64d-cb65a9529625-istio-token" pod "a07ed650-5c07-4440-b64d-cb65a9529625" (UID: "a07ed650-5c07-4440-b64d-cb65a9529625"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:03.112709 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.112672    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c72b9b9-b934-4279-960c-d69fadadb664-istio-podinfo" pod "7c72b9b9-b934-4279-960c-d69fadadb664" (UID: "7c72b9b9-b934-4279-960c-d69fadadb664"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:03.113683 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.113649    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-istio-envoy" pod "7c72b9b9-b934-4279-960c-d69fadadb664" (UID: "7c72b9b9-b934-4279-960c-d69fadadb664"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.115652 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.115617    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07ed650-5c07-4440-b64d-cb65a9529625-kube-api-access-xsrk8" pod "a07ed650-5c07-4440-b64d-cb65a9529625" (UID: "a07ed650-5c07-4440-b64d-cb65a9529625"). InnerVolumeSpecName "kube-api-access-xsrk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:03.211267 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211062    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.211267 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211181    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.211267 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211271    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.211592 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211300    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.211592 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211334    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.211592 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211389    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.211592 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211490    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgp4f\" (UniqueName: \"kubernetes.io/projected/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-kube-api-access-hgp4f\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.211592 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211520    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.211592 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211571    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.211592 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211581    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.211979 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211622    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.211979 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211798    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.211979 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211819    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.211979 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211940    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsrk8\" (UniqueName: \"kubernetes.io/projected/a07ed650-5c07-4440-b64d-cb65a9529625-kube-api-access-xsrk8\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.211979 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211959    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07ed650-5c07-4440-b64d-cb65a9529625-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.211979 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211973    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c72b9b9-b934-4279-960c-d69fadadb664-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.212338 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.211987    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c72b9b9-b934-4279-960c-d69fadadb664-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.212338 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.212000    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07ed650-5c07-4440-b64d-cb65a9529625-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.212338 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.212013    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07ed650-5c07-4440-b64d-cb65a9529625-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.212338 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.212025    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c72b9b9-b934-4279-960c-d69fadadb664-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.212563 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.212407    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.214346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.214321    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.214553 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.214535    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.235426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.235372    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgp4f\" (UniqueName: \"kubernetes.io/projected/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-kube-api-access-hgp4f\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.235534 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.235445    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.381581 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.381543    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.495835    4550 generic.go:356] "Generic (PLEG): container finished" podID="a07ed650-5c07-4440-b64d-cb65a9529625" containerID="9b537ee11bf475283e067cdbaa66a2cac75bf40ec171a77e6ef3e1d4d8b0b437" exitCode=0
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.495901    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.495911    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq" event={"ID":"a07ed650-5c07-4440-b64d-cb65a9529625","Type":"ContainerDied","Data":"9b537ee11bf475283e067cdbaa66a2cac75bf40ec171a77e6ef3e1d4d8b0b437"}
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.495973    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq" event={"ID":"a07ed650-5c07-4440-b64d-cb65a9529625","Type":"ContainerDied","Data":"feeb7946778e2df4211ff6ab51e9abacf0e53ab204c8f34641afeeaf02c9c5e1"}
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.496006    4550 scope.go:122] "RemoveContainer" containerID="9b537ee11bf475283e067cdbaa66a2cac75bf40ec171a77e6ef3e1d4d8b0b437"
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.498242    4550 generic.go:356] "Generic (PLEG): container finished" podID="7c72b9b9-b934-4279-960c-d69fadadb664" containerID="28f101b46402fa9ee6400a5e40a8254b1cacfe618b54879cee5f349aa9c06a57" exitCode=0
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.498315    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8" event={"ID":"7c72b9b9-b934-4279-960c-d69fadadb664","Type":"ContainerDied","Data":"28f101b46402fa9ee6400a5e40a8254b1cacfe618b54879cee5f349aa9c06a57"}
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.498349    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8" event={"ID":"7c72b9b9-b934-4279-960c-d69fadadb664","Type":"ContainerDied","Data":"243b7938b8491486f92bcf850f6544f92ab69bfac9df96a87c7941a8714bbabf"}
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.498432    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.508988    4550 generic.go:356] "Generic (PLEG): container finished" podID="b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" containerID="6f3ee1dcbbb8e825cdc92cfb70a9856c368824bf45f96399e1181b71728a6058" exitCode=0
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.509128    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg" event={"ID":"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7","Type":"ContainerDied","Data":"6f3ee1dcbbb8e825cdc92cfb70a9856c368824bf45f96399e1181b71728a6058"}
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.509225    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg" event={"ID":"b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7","Type":"ContainerDied","Data":"d0eb07e481b259e8ce8aa82673c3de577305c22adb4d26e900546bd25def208b"}
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.509308    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.522615    4550 scope.go:122] "RemoveContainer" containerID="9b537ee11bf475283e067cdbaa66a2cac75bf40ec171a77e6ef3e1d4d8b0b437"
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:10:03.523034    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b537ee11bf475283e067cdbaa66a2cac75bf40ec171a77e6ef3e1d4d8b0b437\": container with ID starting with 9b537ee11bf475283e067cdbaa66a2cac75bf40ec171a77e6ef3e1d4d8b0b437 not found: ID does not exist" containerID="9b537ee11bf475283e067cdbaa66a2cac75bf40ec171a77e6ef3e1d4d8b0b437"
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.523069    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b537ee11bf475283e067cdbaa66a2cac75bf40ec171a77e6ef3e1d4d8b0b437"} err="failed to get container status \"9b537ee11bf475283e067cdbaa66a2cac75bf40ec171a77e6ef3e1d4d8b0b437\": rpc error: code = NotFound desc = could not find container \"9b537ee11bf475283e067cdbaa66a2cac75bf40ec171a77e6ef3e1d4d8b0b437\": container with ID starting with 9b537ee11bf475283e067cdbaa66a2cac75bf40ec171a77e6ef3e1d4d8b0b437 not found: ID does not exist"
Oct 29 22:10:03.524340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.523095    4550 scope.go:122] "RemoveContainer" containerID="28f101b46402fa9ee6400a5e40a8254b1cacfe618b54879cee5f349aa9c06a57"
Oct 29 22:10:03.544790 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.544771    4550 scope.go:122] "RemoveContainer" containerID="28f101b46402fa9ee6400a5e40a8254b1cacfe618b54879cee5f349aa9c06a57"
Oct 29 22:10:03.545131 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:10:03.545105    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f101b46402fa9ee6400a5e40a8254b1cacfe618b54879cee5f349aa9c06a57\": container with ID starting with 28f101b46402fa9ee6400a5e40a8254b1cacfe618b54879cee5f349aa9c06a57 not found: ID does not exist" containerID="28f101b46402fa9ee6400a5e40a8254b1cacfe618b54879cee5f349aa9c06a57"
Oct 29 22:10:03.545226 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.545196    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f101b46402fa9ee6400a5e40a8254b1cacfe618b54879cee5f349aa9c06a57"} err="failed to get container status \"28f101b46402fa9ee6400a5e40a8254b1cacfe618b54879cee5f349aa9c06a57\": rpc error: code = NotFound desc = could not find container \"28f101b46402fa9ee6400a5e40a8254b1cacfe618b54879cee5f349aa9c06a57\": container with ID starting with 28f101b46402fa9ee6400a5e40a8254b1cacfe618b54879cee5f349aa9c06a57 not found: ID does not exist"
Oct 29 22:10:03.545226 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.545223    4550 scope.go:122] "RemoveContainer" containerID="6f3ee1dcbbb8e825cdc92cfb70a9856c368824bf45f96399e1181b71728a6058"
Oct 29 22:10:03.563709 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.561744    4550 scope.go:122] "RemoveContainer" containerID="6f3ee1dcbbb8e825cdc92cfb70a9856c368824bf45f96399e1181b71728a6058"
Oct 29 22:10:03.563709 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:10:03.563603    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f3ee1dcbbb8e825cdc92cfb70a9856c368824bf45f96399e1181b71728a6058\": container with ID starting with 6f3ee1dcbbb8e825cdc92cfb70a9856c368824bf45f96399e1181b71728a6058 not found: ID does not exist" containerID="6f3ee1dcbbb8e825cdc92cfb70a9856c368824bf45f96399e1181b71728a6058"
Oct 29 22:10:03.563709 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.563641    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f3ee1dcbbb8e825cdc92cfb70a9856c368824bf45f96399e1181b71728a6058"} err="failed to get container status \"6f3ee1dcbbb8e825cdc92cfb70a9856c368824bf45f96399e1181b71728a6058\": rpc error: code = NotFound desc = could not find container \"6f3ee1dcbbb8e825cdc92cfb70a9856c368824bf45f96399e1181b71728a6058\": container with ID starting with 6f3ee1dcbbb8e825cdc92cfb70a9856c368824bf45f96399e1181b71728a6058 not found: ID does not exist"
Oct 29 22:10:03.606056 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.603797    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"]
Oct 29 22:10:03.606837 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.606804    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-nxfmg"]
Oct 29 22:10:03.653153 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.653100    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"]
Oct 29 22:10:03.660931 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.660903    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-6rdqq"]
Oct 29 22:10:03.700481 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.700300    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"]
Oct 29 22:10:03.708198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.708125    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-hk8t8"]
Oct 29 22:10:03.748615 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.748589    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"]
Oct 29 22:10:03.760810 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.760772    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:03.760939 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.760867    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:03.760939 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:03.760910    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:04.011473 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.011368    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c72b9b9-b934-4279-960c-d69fadadb664" path="/var/lib/kubelet/pods/7c72b9b9-b934-4279-960c-d69fadadb664/volumes"
Oct 29 22:10:04.012214 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.012189    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07ed650-5c07-4440-b64d-cb65a9529625" path="/var/lib/kubelet/pods/a07ed650-5c07-4440-b64d-cb65a9529625/volumes"
Oct 29 22:10:04.012891 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.012870    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7" path="/var/lib/kubelet/pods/b3bd4c5a-b9d8-4c9d-8dee-4114edb6ffa7/volumes"
Oct 29 22:10:04.013441 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.013421    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"]
Oct 29 22:10:04.019160 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.019122    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.050227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.050203    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-edc2d3-openshift-default-dockercfg-2pkvj\""
Oct 29 22:10:04.057471 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.057429    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:04.090286 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.090251    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"]
Oct 29 22:10:04.193824 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.193790    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.194331 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.193836    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.194331 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.194001    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.194331 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.194043    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.194331 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.194263    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.194331 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.194314    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.194543 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.194517    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh7zv\" (UniqueName: \"kubernetes.io/projected/25bb493a-96b9-4ddf-af48-ee7eee126bef-kube-api-access-mh7zv\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.194576 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.194555    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.194608 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.194580    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25bb493a-96b9-4ddf-af48-ee7eee126bef-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.296566 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.296483    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7zv\" (UniqueName: \"kubernetes.io/projected/25bb493a-96b9-4ddf-af48-ee7eee126bef-kube-api-access-mh7zv\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.296566 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.296521    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.296566 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.296537    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25bb493a-96b9-4ddf-af48-ee7eee126bef-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.296850 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.296699    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.296850 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.296735    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.296850 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.296775    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.296850 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.296803    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.297070 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.296927    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.297070 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.296953    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.297070 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.296972    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.297261 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.297209    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.297261 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.297245    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.297461 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.297441    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25bb493a-96b9-4ddf-af48-ee7eee126bef-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.297522 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.297488    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.298924 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.298904    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.299382 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.299359    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.327623 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.327594    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.340440 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.340403    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh7zv\" (UniqueName: \"kubernetes.io/projected/25bb493a-96b9-4ddf-af48-ee7eee126bef-kube-api-access-mh7zv\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.520458 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.520415    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" event={"ID":"96d90e2d-63dd-461e-a0e9-b6d95ce1707e","Type":"ContainerStarted","Data":"165af8c50d56e110d5bd9af319d8877834ab8b716f0d41ab1158d65928a13487"}
Oct 29 22:10:04.520632 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.520519    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" event={"ID":"96d90e2d-63dd-461e-a0e9-b6d95ce1707e","Type":"ContainerStarted","Data":"542ef134dd77f407736fb6f55bdc7ea7039a1b585c3cca9408ab663a4f980380"}
Oct 29 22:10:04.576663 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.576603    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podStartSLOduration=2.576588711 podStartE2EDuration="2.576588711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:04.574947328 +0000 UTC m=+1420.732072837" watchObservedRunningTime="2026-10-29 22:10:04.576588711 +0000 UTC m=+1420.733714209"
Oct 29 22:10:04.636121 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.636087    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:04.999509 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:04.999481    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"]
Oct 29 22:10:05.014417 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:05.013809    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:05.014417 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:05.013901    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:05.014417 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:05.013944    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:05.395767 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:05.395732    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:05.398518 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:05.398464    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:05.398685 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:05.398612    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:05.535742 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:05.535698    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" event={"ID":"25bb493a-96b9-4ddf-af48-ee7eee126bef","Type":"ContainerStarted","Data":"c3da622d4ea8ce35ebd711300c51c6568c59484aab09d40673508f6cb28c51cd"}
Oct 29 22:10:05.535919 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:05.535772    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" event={"ID":"25bb493a-96b9-4ddf-af48-ee7eee126bef","Type":"ContainerStarted","Data":"c23bb6e464a3a0091ef55952e5ce7a9dbd396192456c3bd7693effdcbf216046"}
Oct 29 22:10:05.608902 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:05.608833    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podStartSLOduration=2.608811109 podStartE2EDuration="2.608811109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:05.604693523 +0000 UTC m=+1421.761819022" watchObservedRunningTime="2026-10-29 22:10:05.608811109 +0000 UTC m=+1421.765936607"
Oct 29 22:10:05.637038 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:05.637006    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:06.013168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:06.012944    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:06.384879 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:06.384844    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:06.385183 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:06.385131    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:06.637659 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:06.637587    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:06.637659 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:06.637631    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:06.717939 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:06.717885    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:06.854692 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:06.853619    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:07.139802 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:07.138419    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:07.438231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:07.432955    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:07.438231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:07.433012    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:07.646942 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:07.639580    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:07.646942 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:07.639636    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:07.942775 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:07.942670    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:08.417186 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:08.415442    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:08.417186 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:08.415510    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:08.637716 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:08.637680    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:08.637999 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:08.637969    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:09.317368 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:09.317307    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:09.387091 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:09.387042    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:09.387443 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:09.387397    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:09.638634 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:09.638588    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:09.638866 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:09.638658    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:10.014235 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.012251    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.052820 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.052770    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.084320 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.084272    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.115307 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.115251    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.132310 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.132252    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.190304 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.190254    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.221296 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.221247    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:10:10.221871 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.221248    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.238178 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.235275    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:10:10.255325 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.255283    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:10:10.276885 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.276789    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:10.328164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.321368    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.328164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.322134    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:10:10.405177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.385607    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:10.405177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.385670    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:10.536556 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.536178    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 29 22:10:10.536814 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.536783    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.655164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.645551    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:10.655164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.645608    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:10.655771 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.655720    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded"
Oct 29 22:10:10.757281 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:10.757238    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:11.015340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.015279    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:11.026173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.019807    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:10:11.063303 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.063262    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:11.082581 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.082528    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:10:11.117424 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.103859    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:10:11.127331 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.119992    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:11.127651 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.127588    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:10:11.129390 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.127955    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:10:11.138351 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.137230    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:11.148388 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.138568    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:10:11.175927 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.175897    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:10:11.195634 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.195593    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:11.199774 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.199746    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:10:11.235209 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.229963    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:11.235209 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.231216    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:10:11.235957 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.235935    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:10:11.236136 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.236124    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:10:11.258513 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.255720    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:11.258513 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.256080    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:10:11.431176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.387977    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:10:11.431176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.413568    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:11.431176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.413623    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:11.530206 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.519519    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:11.601166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.577234    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:11.649383 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.649351    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:11.649675 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.649643    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:11.798935 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.798837    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:10:11.799265 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:11.799235    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:10:12.015180 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:10:12.011799    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:10:12.049224 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.045227    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:10:12.096248 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.096112    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.139168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.133820    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.249174 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.247422    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.332363 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.273874    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:10:12.332363 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.311239    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.413106 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.393805    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.459271 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.413635    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.495165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.478395    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.495165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.488409    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.539976 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.539927    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:12.540266 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.540233    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:12.597614 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.597510    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:12.597937 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.597901    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw" podUID="df402fae-12c4-4ed9-9fb1-226a51f66c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.730198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.725587    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.914212 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.821967    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:13.072161 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.938322    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:13.072161 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:12.952239    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:10:13.196229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.166451    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.287253 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.286548    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.287253 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.286635    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:10:13.335180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.312489    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:10:13.335180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.314974    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.335180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.321312    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.414699 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.411243    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.426176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.426100    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.458411 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.458291    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.513168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.480112    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:13.564167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.524538    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:10:13.564167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.524631    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:10:13.564167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.542436    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:10:13.585165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.574731    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.648189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.586978    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:13.648189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.635308    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:13.648189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.589927    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.757260 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.745770    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:13.757260 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.745828    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:13.757260 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:13.746101    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:14.094331 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.094290    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:10:14.183262 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.183221    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.291339 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.291289    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.330162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.315295    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.330162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.315858    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.330162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.316740    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:10:14.330162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.328949    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.371243 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.370409    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"]
Oct 29 22:10:14.384914 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.383226    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:14.384914 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.383285    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:14.432167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.428281    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.432167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.428668    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.482201 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.482165    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:14.482641 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.482612    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.483130 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.483109    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:10:14.556810 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.555328    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.556810 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.555762    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:10:14.642317 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.642214    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.642546 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.642240    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:14.642724 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.642704    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:14.723245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.723195    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"]
Oct 29 22:10:14.912105 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:14.912021    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:10:15.104880 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.104825    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:15.131164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.109089    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:10:15.259352 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.259255    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:10:15.299978 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.299937    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:15.363175 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.342522    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:10:15.363175 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.342876    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:10:15.363175 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.354280    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:10:15.421432 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.421397    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:15.426192 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.421846    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:15.462497 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.443658    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:15.462497 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.443979    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:10:15.495209 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.486630    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:15.667165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.642282    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:15.667660 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.667305    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:15.667660 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.642709    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:15.667787 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.642409    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:15.780167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.772127    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:15.922221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:15.916395    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:16.037164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.023661    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" podUID="93b4187a-1551-4cf8-b807-d057baf03920" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.134655 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.124246    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:10:16.261515 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.261339    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.308122 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.308069    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.332193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.318116    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.355433 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.355374    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.358092 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.358045    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.375221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.375185    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:10:16.431007 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.430953    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:16.431229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.431031    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:16.451177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.450864    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:10:16.470195 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.465246    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:10:16.511199 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.486878    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:10:16.646914 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.646869    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.661177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.660449    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:16.661177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.660511    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:16.676713 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.674236    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.676713 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:16.674576    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.208645 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.208592    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.310700 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.310648    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.311316 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.311285    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:10:17.360545 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.360498    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.361201 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.361169    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:10:17.390575 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.378383    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:10:17.399617 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.399577    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:17.399903 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.399871    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:17.467572 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.458506    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.468225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.468180    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.491492 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.489252    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.637191 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.637136    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:17.638922 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.638888    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:17.667883 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.653453    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.668934 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.668907    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:10:17.689186 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.681383    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.689186 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.681851    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:10:17.851328 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:17.851281    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:18.062169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.037263    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:18.217219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.211280    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:18.312099 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.287757    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"]
Oct 29 22:10:18.313821 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.313792    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.333914 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.333863    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:10:18.346047 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.346015    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f363a3-openshift-default-dockercfg-6v2wv\""
Oct 29 22:10:18.448200 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.334947    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:18.448200 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.381606    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:10:18.448200 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.381965    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:18.448200 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.386265    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:18.448200 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.386309    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:18.448200 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.386859    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:18.448200 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.390206    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:18.464275 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.464234    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:18.480411 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.480321    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:18.496170 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.492317    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:10:18.505275 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.505221    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"]
Oct 29 22:10:18.571344 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.571300    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:18.575262 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.575215    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:18.599782 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.581109    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.617931 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.617885    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.618315 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.618249    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.619908 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.618498    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.627167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.624020    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.636361 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.636326    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.672168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.644401    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvfds\" (UniqueName: \"kubernetes.io/projected/73cee1bc-536f-4cc5-8874-7409ae9f739c-kube-api-access-rvfds\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.672560 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.672526    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.725169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.699047    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73cee1bc-536f-4cc5-8874-7409ae9f739c-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.743288 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.673078    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:10:18.769187 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.690291    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:18.769675 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.690331    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:10:18.770188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.690352    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:10:18.796172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.647249    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:18.796499 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.796456    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:18.861172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.856483    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.861172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.856550    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.861172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.856609    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.861172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.856681    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.861172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.856753    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvfds\" (UniqueName: \"kubernetes.io/projected/73cee1bc-536f-4cc5-8874-7409ae9f739c-kube-api-access-rvfds\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.861172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.856819    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.861172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.856960    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73cee1bc-536f-4cc5-8874-7409ae9f739c-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.861172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.857014    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.861172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.857057    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.861172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.857574    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.861172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.859385    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.861172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.859942    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.861172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.860630    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73cee1bc-536f-4cc5-8874-7409ae9f739c-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:18.861172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:18.860897    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:19.145224 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.104720    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:19.153167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.149957    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:19.225539 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.172583    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:19.225539 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.222585    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.257357 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.257321    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:19.362387 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.362348    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"]
Oct 29 22:10:19.425230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.405255    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.441213 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.441135    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:10:19.441720 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.441689    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:10:19.442236 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.442214    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:10:19.479556 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.453319    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:19.479556 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.453383    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:19.479556 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.455341    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.487182 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.485815    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.513815 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.513771    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:10:19.537262 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.536198    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.599164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.595940    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvfds\" (UniqueName: \"kubernetes.io/projected/73cee1bc-536f-4cc5-8874-7409ae9f739c-kube-api-access-rvfds\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:19.674169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.642193    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"]
Oct 29 22:10:19.675065 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.675034    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:19.676372 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.646314    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:19.702306 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.702267    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:19.702599 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.702572    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:19.743624 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.707280    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:19.743624 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.707334    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" podUID="a26797f1-f34c-44e3-86e9-dd0ec24af003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.743624 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.714246    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:19.743624 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.714299    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.771188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.744885    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:19.771188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.744948    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9089a59-c618-4d5c-8957-9d095dd04959-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:19.771188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.745073    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:19.771188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.745104    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2846k\" (UniqueName: \"kubernetes.io/projected/a9089a59-c618-4d5c-8957-9d095dd04959-kube-api-access-2846k\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:19.771188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.745250    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:19.771188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.745278    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9089a59-c618-4d5c-8957-9d095dd04959-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:19.771188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.745318    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9089a59-c618-4d5c-8957-9d095dd04959-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:19.771188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.745581    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:19.771188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.745725    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:19.851169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.788373    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.809924    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.809954    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.874095    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.874710    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.132005    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.132063    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.132105    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9089a59-c618-4d5c-8957-9d095dd04959-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.132246    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.132278    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2846k\" (UniqueName: \"kubernetes.io/projected/a9089a59-c618-4d5c-8957-9d095dd04959-kube-api-access-2846k\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.132407    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.132436    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9089a59-c618-4d5c-8957-9d095dd04959-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.132481    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9089a59-c618-4d5c-8957-9d095dd04959-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.133447    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9089a59-c618-4d5c-8957-9d095dd04959-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.875246    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.133740    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:19.918700    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.144242    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.148022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.144823    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.219457 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.215437    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.254193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.242198    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:10:20.254193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.250800    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2846k\" (UniqueName: \"kubernetes.io/projected/a9089a59-c618-4d5c-8957-9d095dd04959-kube-api-access-2846k\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.313290 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.313253    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9089a59-c618-4d5c-8957-9d095dd04959-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.397567 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.397535    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 29 22:10:20.397830 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.397801    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 29 22:10:20.407272 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.407222    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.447210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.447163    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:10:20.449562 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.447422    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.450883 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.447460    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:10:20.470328 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.470254    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.490178 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.489248    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.517255 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.516233    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.521956 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.521730    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.539119 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.537311    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.680550 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.675970    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:20.688564 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.688223    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.714179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.707661    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:20.714179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.707737    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:20.773041 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.767833    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9089a59-c618-4d5c-8957-9d095dd04959-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:20.773571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:20.773115    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:21.171165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.143794    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" podUID="93b4187a-1551-4cf8-b807-d057baf03920" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.234235 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.197640    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.234235 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.199888    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.234235 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.200284    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.404082 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.403366    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"]
Oct 29 22:10:21.404267 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.404198    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.417301 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.417258    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.498357 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.454269    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:10:21.498357 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.459642    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:10:21.498357 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.491295    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:10:21.498357 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.494882    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:10:21.560167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.507925    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:10:21.560167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.520007    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.560167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.559001    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:10:21.583224 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.580274    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.714168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.710637    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" podUID="93b4187a-1551-4cf8-b807-d057baf03920" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.777167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.774741    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:21.959408 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.959321    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:21.989538 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:21.989494    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:22.016166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.003419    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:22.016166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.003580    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:22.016166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.003677    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:22.260426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.260291    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:10:22.287166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.261086    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:10:22.291379 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.261007    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:22.405549 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.405497    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:22.427873 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.427821    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:10:22.487306 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.487262    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:22.487578 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.487555    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:22.487987 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.487255    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:10:22.488584 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.487288    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:10:22.508541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.508503    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:10:22.509103 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.509073    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 29 22:10:22.555225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.526429    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:22.555225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.529567    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:22.574103 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.574063    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:22.579166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.575775    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8" event={"ID":"73cee1bc-536f-4cc5-8874-7409ae9f739c","Type":"ContainerStarted","Data":"14653c3763b4a6583a5b878643b69a432a3bad9b70287f2915a97b2b162395eb"}
Oct 29 22:10:22.659626 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.659588    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:22.659904 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.659868    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:22.715973 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:22.715254    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Oct 29 22:10:23.182169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.166600    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:23.261989 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.260988    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:10:23.295675 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.295077    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:23.295675 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.295445    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.348167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.333346    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.348167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.333724    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:10:23.433185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.433114    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.450164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.435539    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:10:23.491400 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.491294    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:10:23.492019 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.491986    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.527182 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.517291    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.527182 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.518066    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.560803 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.528378    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.564205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.562508    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.642969 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.611778    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:10:23.796262 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.796166    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:10:23.819672 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.819628    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:10:23.846926 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:23.846319    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 29 22:10:24.546167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.402060    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.546167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.402423    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:24.546167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.545806    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:24.581305 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.581260    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.581944 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.581912    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:24.582361 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.582337    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:24.638189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.618737    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:24.638189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.634860    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.638189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.635175    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:24.638189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.635378    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.638189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.635539    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.638661 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.638639    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.803689 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.803577    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:24.803947 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.803920    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:24.804096 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.804076    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:24.992218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:24.980955    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"]
Oct 29 22:10:25.026327 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:25.024420    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:10:25.076229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:25.076200    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:10:25.121447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:25.121417    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:10:25.349373 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:25.347525    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:10:25.459438 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:25.458173    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:10:25.611821 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:25.562500    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:25.611821 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:25.610302    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:25.654068 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:25.654039    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:10:25.658791 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:25.658763    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:10:25.663078 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:25.663046    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:10:25.693224 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:25.678245    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:10:25.693224 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:25.678400    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:10:25.693224 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:25.678465    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:10:25.720409 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:25.720104    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:26.078753 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:26.078718    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:10:26.099891 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:26.099696    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:26.127815 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:26.127774    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:26.301493 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:26.301394    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85" event={"ID":"a9089a59-c618-4d5c-8957-9d095dd04959","Type":"ContainerStarted","Data":"996d631e108eb1ec262814260d127e9905e82b9eb717ab43a389f9b2432dffec"}
Oct 29 22:10:26.301808 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:26.301785    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:10:26.301978 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:26.301967    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:10:26.302110 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:26.302100    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:10:26.310167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:26.302250    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:10:26.312848 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:26.312827    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:10:26.313015 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:26.312994    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8" event={"ID":"73cee1bc-536f-4cc5-8874-7409ae9f739c","Type":"ContainerStarted","Data":"49dadb9142d5a592356706cd637ec35879e0ee3284110a9746be095fd174d9be"}
Oct 29 22:10:26.313186 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:26.313133    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:10:26.452655 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:26.452537    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8" podStartSLOduration=8.452515192 podStartE2EDuration="8.452515192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:26.40040649 +0000 UTC m=+1442.557531989" watchObservedRunningTime="2026-10-29 22:10:26.452515192 +0000 UTC m=+1442.609640694"
Oct 29 22:10:26.881172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:26.877886    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:26.881172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:26.878078    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:10:26.881172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:26.878134    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8" podUID="73cee1bc-536f-4cc5-8874-7409ae9f739c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:10:27.221798 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:27.214828    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85" event={"ID":"a9089a59-c618-4d5c-8957-9d095dd04959","Type":"ContainerStarted","Data":"fb71b5a29fc96f1e8ce9853c914d790ba0c389ae742053b8968cf25e7775f1ec"}
Oct 29 22:10:27.293872 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:27.293806    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85" podStartSLOduration=8.293783426 podStartE2EDuration="8.293783426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:27.260753583 +0000 UTC m=+1443.417879083" watchObservedRunningTime="2026-10-29 22:10:27.293783426 +0000 UTC m=+1443.450908923"
Oct 29 22:10:27.529225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:27.527033    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:27.653664 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:27.653627    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:27.780951 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:27.780654    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:27.780951 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:27.780776    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:10:27.780951 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:27.780819    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85" podUID="a9089a59-c618-4d5c-8957-9d095dd04959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 29 22:10:27.934166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:27.901608    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:10:27.934166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:27.901666    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8" podUID="73cee1bc-536f-4cc5-8874-7409ae9f739c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:10:27.934166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:27.902379    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:28.040860 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:28.027533    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:28.044566 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:28.044533    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"]
Oct 29 22:10:28.044835 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:28.044819    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"]
Oct 29 22:10:28.045631 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:28.045603    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d" podUID="1be95f05-57d1-4e36-8fd4-96b90281dcee" containerName="istio-proxy" containerID="cri-o://c263da465c0f5c507cc7b14a790d1d8da15be1fddd87c6d1ecbda609b67a557c" gracePeriod=30
Oct 29 22:10:28.045854 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:28.045164    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy" containerID="cri-o://165af8c50d56e110d5bd9af319d8877834ab8b716f0d41ab1158d65928a13487" gracePeriod=30
Oct 29 22:10:28.427169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:28.425831    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:28.641167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:28.640731    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:28.889377 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:28.876099    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:10:28.889377 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:28.876182    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85" podUID="a9089a59-c618-4d5c-8957-9d095dd04959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 29 22:10:28.937370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:28.922867    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:10:28.937370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:28.922927    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8" podUID="73cee1bc-536f-4cc5-8874-7409ae9f739c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:10:29.788231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:29.781528    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:10:29.788231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:29.781588    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85" podUID="a9089a59-c618-4d5c-8957-9d095dd04959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 29 22:10:29.883169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:29.877518    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:10:29.883169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:29.877579    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8" podUID="73cee1bc-536f-4cc5-8874-7409ae9f739c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:10:30.438626 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:30.438587    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:30.774810 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:30.774696    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:10:30.775108 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:30.775059    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85" podUID="a9089a59-c618-4d5c-8957-9d095dd04959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 29 22:10:30.879598 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:30.879560    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:10:30.879904 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:30.879838    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8" podUID="73cee1bc-536f-4cc5-8874-7409ae9f739c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:10:31.776980 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:31.776937    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:10:31.777730 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:31.777609    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85" podUID="a9089a59-c618-4d5c-8957-9d095dd04959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 29 22:10:31.897346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:31.897274    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:31.898276 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:31.898252    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:31.899437 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:31.899413    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:10:32.534786 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:32.534743    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:32.781573 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:32.781536    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:10:32.782971 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:32.782931    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85" podUID="a9089a59-c618-4d5c-8957-9d095dd04959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 29 22:10:33.270838 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.266151    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"]
Oct 29 22:10:33.291609 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.285985    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.321708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.321669    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"]
Oct 29 22:10:33.385671 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.385129    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"]
Oct 29 22:10:33.388802 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.388574    4550 generic.go:356] "Generic (PLEG): container finished" podID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerID="165af8c50d56e110d5bd9af319d8877834ab8b716f0d41ab1158d65928a13487" exitCode=0
Oct 29 22:10:33.388802 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.388732    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" event={"ID":"96d90e2d-63dd-461e-a0e9-b6d95ce1707e","Type":"ContainerDied","Data":"165af8c50d56e110d5bd9af319d8877834ab8b716f0d41ab1158d65928a13487"}
Oct 29 22:10:33.389661 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.389352    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"]
Oct 29 22:10:33.393693 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.392517    4550 generic.go:356] "Generic (PLEG): container finished" podID="1be95f05-57d1-4e36-8fd4-96b90281dcee" containerID="c263da465c0f5c507cc7b14a790d1d8da15be1fddd87c6d1ecbda609b67a557c" exitCode=0
Oct 29 22:10:33.393693 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.393633    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d" event={"ID":"1be95f05-57d1-4e36-8fd4-96b90281dcee","Type":"ContainerDied","Data":"c263da465c0f5c507cc7b14a790d1d8da15be1fddd87c6d1ecbda609b67a557c"}
Oct 29 22:10:33.418923 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.418840    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:10:33.428284 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.428256    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:33.456817 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.455919    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b9de66-29e2-4895-9ebe-b006b55614c2-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.456817 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.455995    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.456817 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.456043    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.456817 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.456080    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.456817 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.456109    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.456817 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.456183    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.456817 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.456217    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.456817 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.456300    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.456817 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.456316    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9m4d\" (UniqueName: \"kubernetes.io/projected/b8b9de66-29e2-4895-9ebe-b006b55614c2-kube-api-access-g9m4d\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.556954 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.556924    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-token\" (UniqueName: \"kubernetes.io/projected/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-token\") pod \"1be95f05-57d1-4e36-8fd4-96b90281dcee\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") "
Oct 29 22:10:33.557134 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.557040    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1be95f05-57d1-4e36-8fd4-96b90281dcee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be95f05-57d1-4e36-8fd4-96b90281dcee-istiod-ca-cert\") pod \"1be95f05-57d1-4e36-8fd4-96b90281dcee\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") "
Oct 29 22:10:33.557763 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.557214    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be95f05-57d1-4e36-8fd4-96b90281dcee-kube-api-access-rtzmt\" (UniqueName: \"kubernetes.io/projected/1be95f05-57d1-4e36-8fd4-96b90281dcee-kube-api-access-rtzmt\") pod \"1be95f05-57d1-4e36-8fd4-96b90281dcee\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") "
Oct 29 22:10:33.557763 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.557281    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-podinfo\") pod \"1be95f05-57d1-4e36-8fd4-96b90281dcee\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") "
Oct 29 22:10:33.557763 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.557715    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be95f05-57d1-4e36-8fd4-96b90281dcee-istiod-ca-cert" pod "1be95f05-57d1-4e36-8fd4-96b90281dcee" (UID: "1be95f05-57d1-4e36-8fd4-96b90281dcee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:33.558097 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.558078    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-workload-certs\") pod \"1be95f05-57d1-4e36-8fd4-96b90281dcee\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") "
Oct 29 22:10:33.558242 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.558225    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-workload-socket\") pod \"1be95f05-57d1-4e36-8fd4-96b90281dcee\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") "
Oct 29 22:10:33.558392 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.558334    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-credential-socket\") pod \"1be95f05-57d1-4e36-8fd4-96b90281dcee\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") "
Oct 29 22:10:33.558392 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.558367    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-data\") pod \"1be95f05-57d1-4e36-8fd4-96b90281dcee\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") "
Oct 29 22:10:33.558523 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.558440    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-envoy\") pod \"1be95f05-57d1-4e36-8fd4-96b90281dcee\" (UID: \"1be95f05-57d1-4e36-8fd4-96b90281dcee\") "
Oct 29 22:10:33.558804 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.558783    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.558896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.558847    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.558955 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.558919    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.559010 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.558971    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.559106 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.559092    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.559325 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.559308    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.559478 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.559463    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.559549 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.559497    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9m4d\" (UniqueName: \"kubernetes.io/projected/b8b9de66-29e2-4895-9ebe-b006b55614c2-kube-api-access-g9m4d\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.559717 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.559703    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b9de66-29e2-4895-9ebe-b006b55614c2-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.559818 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.559803    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be95f05-57d1-4e36-8fd4-96b90281dcee-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.560223 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.560196    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be95f05-57d1-4e36-8fd4-96b90281dcee-kube-api-access-rtzmt" pod "1be95f05-57d1-4e36-8fd4-96b90281dcee" (UID: "1be95f05-57d1-4e36-8fd4-96b90281dcee"). InnerVolumeSpecName "kube-api-access-rtzmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:33.560578 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.560547    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b9de66-29e2-4895-9ebe-b006b55614c2-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.560953 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.560885    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.561351 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.561315    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.562705 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.562394    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.563796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.563467    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-podinfo" pod "1be95f05-57d1-4e36-8fd4-96b90281dcee" (UID: "1be95f05-57d1-4e36-8fd4-96b90281dcee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:33.563796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.563474    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-data" pod "1be95f05-57d1-4e36-8fd4-96b90281dcee" (UID: "1be95f05-57d1-4e36-8fd4-96b90281dcee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:33.563796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.563628    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-workload-socket" pod "1be95f05-57d1-4e36-8fd4-96b90281dcee" (UID: "1be95f05-57d1-4e36-8fd4-96b90281dcee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:33.564274 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.564238    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-workload-certs" pod "1be95f05-57d1-4e36-8fd4-96b90281dcee" (UID: "1be95f05-57d1-4e36-8fd4-96b90281dcee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:33.566100 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.566067    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-credential-socket" pod "1be95f05-57d1-4e36-8fd4-96b90281dcee" (UID: "1be95f05-57d1-4e36-8fd4-96b90281dcee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:33.566242 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.566098    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-envoy" pod "1be95f05-57d1-4e36-8fd4-96b90281dcee" (UID: "1be95f05-57d1-4e36-8fd4-96b90281dcee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:33.566624 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.566531    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.569690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.569668    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.571567 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.571547    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-token" pod "1be95f05-57d1-4e36-8fd4-96b90281dcee" (UID: "1be95f05-57d1-4e36-8fd4-96b90281dcee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:33.573588 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.573564    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.581969 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.581952    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:33.592385 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.592363    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.595215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.595190    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9m4d\" (UniqueName: \"kubernetes.io/projected/b8b9de66-29e2-4895-9ebe-b006b55614c2-kube-api-access-g9m4d\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.661512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.661434    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtzmt\" (UniqueName: \"kubernetes.io/projected/1be95f05-57d1-4e36-8fd4-96b90281dcee-kube-api-access-rtzmt\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.661512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.661462    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.661512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.661472    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.661512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.661480    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.661512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.661488    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.661512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.661497    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.661512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.661504    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.661512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.661512    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be95f05-57d1-4e36-8fd4-96b90281dcee-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.665388 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.665359    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:33.708014 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.707977    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:33.762898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.762867    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-workload-socket\") pod \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") "
Oct 29 22:10:33.763060 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.762914    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-podinfo\") pod \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") "
Oct 29 22:10:33.763060 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.763014    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-data\") pod \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") "
Oct 29 22:10:33.763130 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.763079    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-token\" (UniqueName: \"kubernetes.io/projected/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-token\") pod \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") "
Oct 29 22:10:33.763191 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.763138    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-workload-certs\") pod \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") "
Oct 29 22:10:33.763227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.763207    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-credential-socket\") pod \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") "
Oct 29 22:10:33.763298 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.763281    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istiod-ca-cert\") pod \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") "
Oct 29 22:10:33.763517 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.763367    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-kube-api-access-hgp4f\" (UniqueName: \"kubernetes.io/projected/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-kube-api-access-hgp4f\") pod \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") "
Oct 29 22:10:33.763631 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.763530    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-data" pod "96d90e2d-63dd-461e-a0e9-b6d95ce1707e" (UID: "96d90e2d-63dd-461e-a0e9-b6d95ce1707e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:33.763631 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.763595    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-envoy\") pod \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\" (UID: \"96d90e2d-63dd-461e-a0e9-b6d95ce1707e\") "
Oct 29 22:10:33.764065 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.763889    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-workload-socket" pod "96d90e2d-63dd-461e-a0e9-b6d95ce1707e" (UID: "96d90e2d-63dd-461e-a0e9-b6d95ce1707e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:33.764065 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.763899    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-credential-socket" pod "96d90e2d-63dd-461e-a0e9-b6d95ce1707e" (UID: "96d90e2d-63dd-461e-a0e9-b6d95ce1707e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:33.764065 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.763952    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-workload-certs" pod "96d90e2d-63dd-461e-a0e9-b6d95ce1707e" (UID: "96d90e2d-63dd-461e-a0e9-b6d95ce1707e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:33.764065 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.764016    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istiod-ca-cert" pod "96d90e2d-63dd-461e-a0e9-b6d95ce1707e" (UID: "96d90e2d-63dd-461e-a0e9-b6d95ce1707e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:33.764849 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.764824    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.764952 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.764852    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.764952 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.764868    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.764952 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.764882    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.764952 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.764897    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.766273 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.766250    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-podinfo" pod "96d90e2d-63dd-461e-a0e9-b6d95ce1707e" (UID: "96d90e2d-63dd-461e-a0e9-b6d95ce1707e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:33.784989 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.784933    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-envoy" pod "96d90e2d-63dd-461e-a0e9-b6d95ce1707e" (UID: "96d90e2d-63dd-461e-a0e9-b6d95ce1707e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:33.785983 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.785950    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-kube-api-access-hgp4f" pod "96d90e2d-63dd-461e-a0e9-b6d95ce1707e" (UID: "96d90e2d-63dd-461e-a0e9-b6d95ce1707e"). InnerVolumeSpecName "kube-api-access-hgp4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:33.794117 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.794058    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-token" pod "96d90e2d-63dd-461e-a0e9-b6d95ce1707e" (UID: "96d90e2d-63dd-461e-a0e9-b6d95ce1707e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:33.796413 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.794433    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:33.796413 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.795733    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:33.800043 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.799734    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:10:33.870445 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.867180    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.870445 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.867214    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.870445 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.867230    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgp4f\" (UniqueName: \"kubernetes.io/projected/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-kube-api-access-hgp4f\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.870445 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.867244    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96d90e2d-63dd-461e-a0e9-b6d95ce1707e-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.877199 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:33.876084    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:34.049518 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.049464    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:34.049685 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.049605    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:34.049685 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.049656    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:34.055775 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.055707    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"]
Oct 29 22:10:34.414208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.412103    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"]
Oct 29 22:10:34.414208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.413190    4550 cpu_manager.go:414] "containerMap: removing container" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy"
Oct 29 22:10:34.414208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.413211    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy"
Oct 29 22:10:34.414208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.413252    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1be95f05-57d1-4e36-8fd4-96b90281dcee" containerName="istio-proxy"
Oct 29 22:10:34.414208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.413261    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be95f05-57d1-4e36-8fd4-96b90281dcee" containerName="istio-proxy"
Oct 29 22:10:34.414208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.413483    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" containerName="istio-proxy"
Oct 29 22:10:34.414208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.413504    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be95f05-57d1-4e36-8fd4-96b90281dcee" containerName="istio-proxy"
Oct 29 22:10:34.421279 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.421251    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"]
Oct 29 22:10:34.421592 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.421474    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.468858 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.468825    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb" event={"ID":"96d90e2d-63dd-461e-a0e9-b6d95ce1707e","Type":"ContainerDied","Data":"542ef134dd77f407736fb6f55bdc7ea7039a1b585c3cca9408ab663a4f980380"}
Oct 29 22:10:34.469038 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.468892    4550 scope.go:122] "RemoveContainer" containerID="165af8c50d56e110d5bd9af319d8877834ab8b716f0d41ab1158d65928a13487"
Oct 29 22:10:34.469112 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.469055    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"
Oct 29 22:10:34.474303 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.474272    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d" event={"ID":"1be95f05-57d1-4e36-8fd4-96b90281dcee","Type":"ContainerDied","Data":"c6276be1b3f8f9f3125a2361ef4296a48744eb49f554e3c7cb42f10986001e95"}
Oct 29 22:10:34.474596 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.474578    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"
Oct 29 22:10:34.486193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.486122    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w" event={"ID":"b8b9de66-29e2-4895-9ebe-b006b55614c2","Type":"ContainerStarted","Data":"fbc7c0e6a7675bfeb4e2a396b7119b3cb0dcc1968f248fc03288c53ac940a124"}
Oct 29 22:10:34.486193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.486189    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w" event={"ID":"b8b9de66-29e2-4895-9ebe-b006b55614c2","Type":"ContainerStarted","Data":"faec6456ca4b6a39fbc833f56f0cbec989b31df4114bb5306aa610510f3a0de3"}
Oct 29 22:10:34.502263 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.502236    4550 scope.go:122] "RemoveContainer" containerID="c263da465c0f5c507cc7b14a790d1d8da15be1fddd87c6d1ecbda609b67a557c"
Oct 29 22:10:34.504498 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.504463    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd31e322-3948-4e25-998f-be538665aa9e-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.504620 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.504522    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.504620 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.504591    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.504823 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.504794    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49mrm\" (UniqueName: \"kubernetes.io/projected/fd31e322-3948-4e25-998f-be538665aa9e-kube-api-access-49mrm\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.504886 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.504854    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd31e322-3948-4e25-998f-be538665aa9e-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.504941 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.504880    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.504941 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.504902    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd31e322-3948-4e25-998f-be538665aa9e-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.504941 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.504924    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.505247 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.505128    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.526662 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.526633    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"]
Oct 29 22:10:34.533787 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.533754    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-glcdb"]
Oct 29 22:10:34.579095 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.579061    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"]
Oct 29 22:10:34.585761 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.585733    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-58v2d"]
Oct 29 22:10:34.607053 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.607023    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd31e322-3948-4e25-998f-be538665aa9e-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.607272 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.607070    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.607272 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.607122    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.607414 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.607393    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49mrm\" (UniqueName: \"kubernetes.io/projected/fd31e322-3948-4e25-998f-be538665aa9e-kube-api-access-49mrm\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.607477 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.607453    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd31e322-3948-4e25-998f-be538665aa9e-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.607538 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.607479    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.607538 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.607502    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd31e322-3948-4e25-998f-be538665aa9e-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.607538 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.607518    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.607700 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.607598    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.607758 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.607734    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.607919 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.607897    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd31e322-3948-4e25-998f-be538665aa9e-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.608553 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.608529    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.609132 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.609109    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.609348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.609330    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.610716 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.610668    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.611885 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.611800    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd31e322-3948-4e25-998f-be538665aa9e-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.635729 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.635662    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w" podStartSLOduration=1.635641155 podStartE2EDuration="1.635641155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:34.626573703 +0000 UTC m=+1450.783699228" watchObservedRunningTime="2026-10-29 22:10:34.635641155 +0000 UTC m=+1450.792766654"
Oct 29 22:10:34.642206 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.642134    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd31e322-3948-4e25-998f-be538665aa9e-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.642826 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.642800    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49mrm\" (UniqueName: \"kubernetes.io/projected/fd31e322-3948-4e25-998f-be538665aa9e-kube-api-access-49mrm\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.666151 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.666107    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:34.770868 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.769443    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:34.892576 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.892546    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"]
Oct 29 22:10:34.921252 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.911446    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:34.926196 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.917483    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"]
Oct 29 22:10:34.956583 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.956551    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-27cda7-serving-cert\""
Oct 29 22:10:34.993672 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:34.993637    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:35.014928 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.014840    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-metrics-certs\") pod \"maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:35.015387 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.015362    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk8xq\" (UniqueName: \"kubernetes.io/projected/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-kube-api-access-lk8xq\") pod \"maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:35.015707 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.015685    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:35.015940 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.015914    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:35.117751 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.117665    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:35.117905 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.117803    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:35.117946 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.117909    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-metrics-certs\") pod \"maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:35.118686 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.118640    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk8xq\" (UniqueName: \"kubernetes.io/projected/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-kube-api-access-lk8xq\") pod \"maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:35.119019 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.118667    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:35.120766 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.120741    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:35.121374 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.121356    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-metrics-certs\") pod \"maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:35.139348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.139310    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"]
Oct 29 22:10:35.155456 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.154021    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:35.156698 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.156669    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk8xq\" (UniqueName: \"kubernetes.io/projected/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-kube-api-access-lk8xq\") pod \"maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:35.157678 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:10:35.157650    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd31e322_3948_4e25_998f_be538665aa9e.slice/crio-8a90233b06a2ff94e0b678b82e4811ea3b17e860787f29af70afaf7e641c6f9e WatchSource:0}: Error finding container 8a90233b06a2ff94e0b678b82e4811ea3b17e860787f29af70afaf7e641c6f9e: Status 404 returned error can't find the container with id 8a90233b06a2ff94e0b678b82e4811ea3b17e860787f29af70afaf7e641c6f9e
Oct 29 22:10:35.161490 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.161457    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:35.161619 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.161533    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:35.161619 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.161575    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:35.282916 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.282886    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:35.539341 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.539300    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg" event={"ID":"fd31e322-3948-4e25-998f-be538665aa9e","Type":"ContainerStarted","Data":"7fb3fa4d762d99e52b943142e4ca83a678c9bc5633419ae3edc8287aa0d6be12"}
Oct 29 22:10:35.539596 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.539574    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg" event={"ID":"fd31e322-3948-4e25-998f-be538665aa9e","Type":"ContainerStarted","Data":"8a90233b06a2ff94e0b678b82e4811ea3b17e860787f29af70afaf7e641c6f9e"}
Oct 29 22:10:35.622023 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.619175    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg" podStartSLOduration=1.619132707 podStartE2EDuration="1.619132707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:35.585197857 +0000 UTC m=+1451.742323383" watchObservedRunningTime="2026-10-29 22:10:35.619132707 +0000 UTC m=+1451.776258206"
Oct 29 22:10:35.709305 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.709213    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:10:35.709561 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.709531    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w" podUID="b8b9de66-29e2-4895-9ebe-b006b55614c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 29 22:10:35.723191 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.721799    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"]
Oct 29 22:10:35.728004 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:10:35.727975    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06aa5d20_0931_4e09_98ef_6b7cf4c1a59d.slice/crio-f5dde086ec241dc72e382b9ebccdd9b66706ed9f6af1272b494d4641a137abe1 WatchSource:0}: Error finding container f5dde086ec241dc72e382b9ebccdd9b66706ed9f6af1272b494d4641a137abe1: Status 404 returned error can't find the container with id f5dde086ec241dc72e382b9ebccdd9b66706ed9f6af1272b494d4641a137abe1
Oct 29 22:10:35.781286 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:35.781240    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:36.156025 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:36.143091    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be95f05-57d1-4e36-8fd4-96b90281dcee" path="/var/lib/kubelet/pods/1be95f05-57d1-4e36-8fd4-96b90281dcee/volumes"
Oct 29 22:10:36.156025 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:36.144094    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d90e2d-63dd-461e-a0e9-b6d95ce1707e" path="/var/lib/kubelet/pods/96d90e2d-63dd-461e-a0e9-b6d95ce1707e/volumes"
Oct 29 22:10:36.557935 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:36.557893    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp" event={"ID":"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d","Type":"ContainerStarted","Data":"d8a85cdd2158e5ceb73a911d742b8e79b7597295b46b0caff5baf72880c63861"}
Oct 29 22:10:36.558127 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:36.557962    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:36.558127 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:36.557985    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp" event={"ID":"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d","Type":"ContainerStarted","Data":"f5dde086ec241dc72e382b9ebccdd9b66706ed9f6af1272b494d4641a137abe1"}
Oct 29 22:10:36.644165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:36.641300    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp" podStartSLOduration=1.957215808 podStartE2EDuration="2.641280833s" totalImagesPullingTime="684.065025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:36.630222621 +0000 UTC m=+1452.787348133" watchObservedRunningTime="2026-10-29 22:10:36.641280833 +0000 UTC m=+1452.798406332"
Oct 29 22:10:36.685227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:36.668610    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:10:36.685227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:36.668694    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w" podUID="b8b9de66-29e2-4895-9ebe-b006b55614c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 29 22:10:36.846733 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:36.846684    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:36.855313 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:36.852058    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:10:36.855313 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:36.852113    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg" podUID="fd31e322-3948-4e25-998f-be538665aa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 22:10:37.669586 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:37.669556    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:10:37.708243 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:37.707217    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w" podUID="b8b9de66-29e2-4895-9ebe-b006b55614c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 29 22:10:37.793168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:37.792254    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:10:37.793168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:37.792327    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg" podUID="fd31e322-3948-4e25-998f-be538665aa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 22:10:37.813668 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:37.811386    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:37.936481 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:37.936388    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:38.693166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:38.668516    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:10:38.693166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:38.668575    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w" podUID="b8b9de66-29e2-4895-9ebe-b006b55614c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 29 22:10:38.733265 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:38.733225    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:38.840718 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:38.835117    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:10:38.840718 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:38.835194    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg" podUID="fd31e322-3948-4e25-998f-be538665aa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 22:10:39.698188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:39.697774    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:10:39.698188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:39.697831    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w" podUID="b8b9de66-29e2-4895-9ebe-b006b55614c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 29 22:10:39.806164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:39.783544    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:10:39.806164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:39.783608    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg" podUID="fd31e322-3948-4e25-998f-be538665aa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 22:10:40.319237 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:40.319195    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" containerID="cri-o://82b0d75de4a3f8bc6d99224e95f838db55c3262182841a111f7d2ad02795a501" gracePeriod=35
Oct 29 22:10:40.556135 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:40.556080    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": dial tcp 10.133.0.76:8000: connect: connection refused"
Oct 29 22:10:40.691167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:40.688873    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:10:40.691167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:40.688956    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w" podUID="b8b9de66-29e2-4895-9ebe-b006b55614c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 29 22:10:40.757650 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:40.751446    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" containerID="cri-o://4440beae683b0a246401270560cc8144dc3de7301f46f3257e57fbe18932e54e" gracePeriod=35
Oct 29 22:10:40.757650 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:40.757532    4550 generic.go:356] "Generic (PLEG): container finished" podID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerID="82b0d75de4a3f8bc6d99224e95f838db55c3262182841a111f7d2ad02795a501" exitCode=0
Oct 29 22:10:40.758391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:40.757793    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" event={"ID":"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963","Type":"ContainerDied","Data":"82b0d75de4a3f8bc6d99224e95f838db55c3262182841a111f7d2ad02795a501"}
Oct 29 22:10:40.758391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:40.758232    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:40.807925 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:40.807887    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:10:40.845547 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:40.845498    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg" podUID="fd31e322-3948-4e25-998f-be538665aa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 22:10:41.131336 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.128121    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:10:41.216215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.215263    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-kube-api-access-qpjkk\" (UniqueName: \"kubernetes.io/projected/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-kube-api-access-qpjkk\") pod \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") "
Oct 29 22:10:41.216215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.215334    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-model-cache\") pod \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") "
Oct 29 22:10:41.216215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.215417    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-dshm\") pod \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") "
Oct 29 22:10:41.216215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.215917    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-home\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-home\") pod \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") "
Oct 29 22:10:41.216215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.216025    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-tls-certs\") pod \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") "
Oct 29 22:10:41.216215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.216107    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-tmp-dir\") pod \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\" (UID: \"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963\") "
Oct 29 22:10:41.221926 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.217042    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-tmp-dir" pod "3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" (UID: "3a90acbd-b3d5-40fe-ac0c-60a84b0dd963"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:41.221926 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.217205    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-model-cache" pod "3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" (UID: "3a90acbd-b3d5-40fe-ac0c-60a84b0dd963"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:41.221926 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.221618    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-dshm" pod "3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" (UID: "3a90acbd-b3d5-40fe-ac0c-60a84b0dd963"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:41.222275 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.222095    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-home" pod "3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" (UID: "3a90acbd-b3d5-40fe-ac0c-60a84b0dd963"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:41.234352 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.233845    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-kube-api-access-qpjkk" pod "3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" (UID: "3a90acbd-b3d5-40fe-ac0c-60a84b0dd963"). InnerVolumeSpecName "kube-api-access-qpjkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:41.241366 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.241305    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-tls-certs" pod "3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" (UID: "3a90acbd-b3d5-40fe-ac0c-60a84b0dd963"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:10:41.324700 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.324662    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpjkk\" (UniqueName: \"kubernetes.io/projected/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-kube-api-access-qpjkk\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:41.324700 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.324697    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-model-cache\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:41.324939 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.324712    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-dshm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:41.324939 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.324724    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-home\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:41.324939 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.324736    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-tls-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:41.324939 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.324748    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963-tmp-dir\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:41.439105 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.438953    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:41.590773 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:10:41.588378    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:10:41.590773 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.590603    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:41.595296 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.595275    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:10:41.642411 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.642378    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-dshm\") pod \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") "
Oct 29 22:10:41.642584 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.642488    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-tls-certs\") pod \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") "
Oct 29 22:10:41.642584 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.642556    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-model-cache\") pod \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") "
Oct 29 22:10:41.642723 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.642705    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-kube-api-access-gv9dc\" (UniqueName: \"kubernetes.io/projected/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-kube-api-access-gv9dc\") pod \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") "
Oct 29 22:10:41.643802 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.642894    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-tmp-dir\") pod \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") "
Oct 29 22:10:41.643802 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.642950    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-home\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-home\") pod \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\" (UID: \"2ec7f0f1-9454-46d6-991e-3dfb6d2645da\") "
Oct 29 22:10:41.643802 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.643013    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-model-cache" pod "2ec7f0f1-9454-46d6-991e-3dfb6d2645da" (UID: "2ec7f0f1-9454-46d6-991e-3dfb6d2645da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:41.643802 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.643189    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-tmp-dir" pod "2ec7f0f1-9454-46d6-991e-3dfb6d2645da" (UID: "2ec7f0f1-9454-46d6-991e-3dfb6d2645da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:41.643802 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.643739    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-model-cache\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:41.643802 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.643757    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-tmp-dir\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:41.645626 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.644345    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-home" pod "2ec7f0f1-9454-46d6-991e-3dfb6d2645da" (UID: "2ec7f0f1-9454-46d6-991e-3dfb6d2645da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:41.645626 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.645571    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-kube-api-access-gv9dc" pod "2ec7f0f1-9454-46d6-991e-3dfb6d2645da" (UID: "2ec7f0f1-9454-46d6-991e-3dfb6d2645da"). InnerVolumeSpecName "kube-api-access-gv9dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:41.646052 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.646023    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-tls-certs" pod "2ec7f0f1-9454-46d6-991e-3dfb6d2645da" (UID: "2ec7f0f1-9454-46d6-991e-3dfb6d2645da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:10:41.646583 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.646538    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-dshm" pod "2ec7f0f1-9454-46d6-991e-3dfb6d2645da" (UID: "2ec7f0f1-9454-46d6-991e-3dfb6d2645da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:41.700177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.698716    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:41.701231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.700882    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:41.702920 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.702898    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:10:41.745876 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.745848    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv9dc\" (UniqueName: \"kubernetes.io/projected/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-kube-api-access-gv9dc\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:41.745876 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.745875    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-home\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:41.746077 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.745890    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-dshm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:41.746077 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.745902    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec7f0f1-9454-46d6-991e-3dfb6d2645da-tls-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:41.776042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.772622    4550 generic.go:356] "Generic (PLEG): container finished" podID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerID="4440beae683b0a246401270560cc8144dc3de7301f46f3257e57fbe18932e54e" exitCode=0
Oct 29 22:10:41.776042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.772757    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" event={"ID":"2ec7f0f1-9454-46d6-991e-3dfb6d2645da","Type":"ContainerDied","Data":"4440beae683b0a246401270560cc8144dc3de7301f46f3257e57fbe18932e54e"}
Oct 29 22:10:41.776042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.772831    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" event={"ID":"2ec7f0f1-9454-46d6-991e-3dfb6d2645da","Type":"ContainerDied","Data":"7b6ac936e14d3d68f3c08ef13e4c39332d8440e48610ca865b3af5cfdfb927e5"}
Oct 29 22:10:41.776042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.772858    4550 scope.go:122] "RemoveContainer" containerID="4440beae683b0a246401270560cc8144dc3de7301f46f3257e57fbe18932e54e"
Oct 29 22:10:41.776042 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.773022    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"
Oct 29 22:10:41.796382 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.795263    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:10:41.796382 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.795302    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg" podUID="fd31e322-3948-4e25-998f-be538665aa9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 22:10:41.796382 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.796272    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"
Oct 29 22:10:41.796382 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.796293    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m" event={"ID":"3a90acbd-b3d5-40fe-ac0c-60a84b0dd963","Type":"ContainerDied","Data":"e8e945df186390f30967cd3587c5591cf8f384ac595c4ce9ba5b6e930184d062"}
Oct 29 22:10:41.804349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.804284    4550 scope.go:122] "RemoveContainer" containerID="4440beae683b0a246401270560cc8144dc3de7301f46f3257e57fbe18932e54e"
Oct 29 22:10:41.804665 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:10:41.804630    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4440beae683b0a246401270560cc8144dc3de7301f46f3257e57fbe18932e54e\": container with ID starting with 4440beae683b0a246401270560cc8144dc3de7301f46f3257e57fbe18932e54e not found: ID does not exist" containerID="4440beae683b0a246401270560cc8144dc3de7301f46f3257e57fbe18932e54e"
Oct 29 22:10:41.804760 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.804677    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4440beae683b0a246401270560cc8144dc3de7301f46f3257e57fbe18932e54e"} err="failed to get container status \"4440beae683b0a246401270560cc8144dc3de7301f46f3257e57fbe18932e54e\": rpc error: code = NotFound desc = could not find container \"4440beae683b0a246401270560cc8144dc3de7301f46f3257e57fbe18932e54e\": container with ID starting with 4440beae683b0a246401270560cc8144dc3de7301f46f3257e57fbe18932e54e not found: ID does not exist"
Oct 29 22:10:41.804760 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.804700    4550 scope.go:122] "RemoveContainer" containerID="82b0d75de4a3f8bc6d99224e95f838db55c3262182841a111f7d2ad02795a501"
Oct 29 22:10:41.837831 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.837803    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"]
Oct 29 22:10:41.846501 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.846473    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m"]
Oct 29 22:10:41.896858 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.896820    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"]
Oct 29 22:10:41.900208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:41.900186    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/distinct-2-w2-a9def7-kserve-68bb48994-cgk5m"]
Oct 29 22:10:42.077103 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:42.077064    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" path="/var/lib/kubelet/pods/2ec7f0f1-9454-46d6-991e-3dfb6d2645da/volumes"
Oct 29 22:10:42.078330 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:42.078304    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" path="/var/lib/kubelet/pods/3a90acbd-b3d5-40fe-ac0c-60a84b0dd963/volumes"
Oct 29 22:10:42.444370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:42.444264    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/premium-w2-a9def7-kserve-cb8565594-9r87m" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:42.610018 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:42.609986    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:10:42.775368 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:42.775296    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:42.775616 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:42.775594    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:42.776256 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:42.776237    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:10:42.948021 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:42.947988    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:44.027401 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.027366    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"]
Oct 29 22:10:44.028237 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.028215    4550 cpu_manager.go:414] "containerMap: removing container" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main"
Oct 29 22:10:44.028237 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.028236    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main"
Oct 29 22:10:44.028412 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.028272    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main"
Oct 29 22:10:44.028412 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.028280    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main"
Oct 29 22:10:44.028511 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.028479    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec7f0f1-9454-46d6-991e-3dfb6d2645da" containerName="main"
Oct 29 22:10:44.028511 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.028507    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a90acbd-b3d5-40fe-ac0c-60a84b0dd963" containerName="main"
Oct 29 22:10:44.033826 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.033803    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.055541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.055517    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"]
Oct 29 22:10:44.063136 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.063112    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a9e67f-openshift-default-dockercfg-kbmzv\""
Oct 29 22:10:44.101556 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.101527    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.101712 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.101586    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.101712 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.101702    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.101825 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.101770    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.101825 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.101799    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdjdp\" (UniqueName: \"kubernetes.io/projected/8f0d50a6-d0cf-44ae-86e2-32558679df0b-kube-api-access-pdjdp\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.101973 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.101950    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.102096 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.102080    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.102171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.102117    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.102171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.102163    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.203962 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.203929    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.204205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.203972    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.204205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.203993    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.204205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.204120    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.204427 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.204280    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.204534 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.204504    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.204534 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.204517    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.204675 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.204627    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.204731 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.204692    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdjdp\" (UniqueName: \"kubernetes.io/projected/8f0d50a6-d0cf-44ae-86e2-32558679df0b-kube-api-access-pdjdp\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.204838 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.204816    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.204924 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.204844    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.205052 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.205030    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.205136 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.205059    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.205345 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.205316    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.206571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.206550    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.206816 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.206799    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.227076 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.227044    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.227420 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.227398    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdjdp\" (UniqueName: \"kubernetes.io/projected/8f0d50a6-d0cf-44ae-86e2-32558679df0b-kube-api-access-pdjdp\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.348297 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.348264    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:44.674578 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.674553    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"]
Oct 29 22:10:44.675512 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:10:44.675487    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0d50a6_d0cf_44ae_86e2_32558679df0b.slice/crio-6717d3d4a1073ee7ec481d141accdef46f4807db448dad80270ee7883cb16546 WatchSource:0}: Error finding container 6717d3d4a1073ee7ec481d141accdef46f4807db448dad80270ee7883cb16546: Status 404 returned error can't find the container with id 6717d3d4a1073ee7ec481d141accdef46f4807db448dad80270ee7883cb16546
Oct 29 22:10:44.678737 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.678703    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:44.678849 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.678782    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:44.678849 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.678827    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:10:44.827661 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.827571    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm" event={"ID":"8f0d50a6-d0cf-44ae-86e2-32558679df0b","Type":"ContainerStarted","Data":"8b4497459c15992a3e0bc3cb6deb10c58c2b682e64971a770d8c0d1a00b50551"}
Oct 29 22:10:44.827661 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.827650    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm" event={"ID":"8f0d50a6-d0cf-44ae-86e2-32558679df0b","Type":"ContainerStarted","Data":"6717d3d4a1073ee7ec481d141accdef46f4807db448dad80270ee7883cb16546"}
Oct 29 22:10:44.876630 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:44.876582    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm" podStartSLOduration=1.876566077 podStartE2EDuration="1.876566077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:44.872689305 +0000 UTC m=+1461.029814807" watchObservedRunningTime="2026-10-29 22:10:44.876566077 +0000 UTC m=+1461.033691574"
Oct 29 22:10:45.254185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:45.254156    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:45.352417 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:45.351362    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:45.353904 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:45.353262    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:10:45.353904 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:45.353333    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm" podUID="8f0d50a6-d0cf-44ae-86e2-32558679df0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 29 22:10:46.348758 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:46.348723    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:10:46.349218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:46.348773    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm" podUID="8f0d50a6-d0cf-44ae-86e2-32558679df0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 29 22:10:47.157688 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:47.157654    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:47.349128 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:47.349081    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:10:47.349685 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:47.349192    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm" podUID="8f0d50a6-d0cf-44ae-86e2-32558679df0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 29 22:10:48.349337 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.349301    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:10:48.349764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.349354    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm" podUID="8f0d50a6-d0cf-44ae-86e2-32558679df0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 29 22:10:48.418907 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.418868    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main" containerID="cri-o://06bd3eb66bf2649d44f36649e534bfd32f3bee597d1fa4bad669f526a5fa015e" gracePeriod=45
Oct 29 22:10:48.431834 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.431794    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main" containerID="cri-o://98453a51df008f8069c330ac05ec7a07e6a634874aef6d40f4084d4a261ba3fc" gracePeriod=45
Oct 29 22:10:48.853486 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.853462    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:10:48.857264 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.857245    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:10:48.915799 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.915715    4550 generic.go:356] "Generic (PLEG): container finished" podID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerID="98453a51df008f8069c330ac05ec7a07e6a634874aef6d40f4084d4a261ba3fc" exitCode=0
Oct 29 22:10:48.915799 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.915760    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"
Oct 29 22:10:48.916089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.915809    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" event={"ID":"62e5ef97-3c39-467a-8bf8-60d549360d47","Type":"ContainerDied","Data":"98453a51df008f8069c330ac05ec7a07e6a634874aef6d40f4084d4a261ba3fc"}
Oct 29 22:10:48.916089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.915879    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr" event={"ID":"62e5ef97-3c39-467a-8bf8-60d549360d47","Type":"ContainerDied","Data":"af7a5b87d7b0fce76571bd2b149c4d7fd97ac711523e1b0275b996c332252c7d"}
Oct 29 22:10:48.916089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.915905    4550 scope.go:122] "RemoveContainer" containerID="98453a51df008f8069c330ac05ec7a07e6a634874aef6d40f4084d4a261ba3fc"
Oct 29 22:10:48.918342 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.918316    4550 generic.go:356] "Generic (PLEG): container finished" podID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerID="06bd3eb66bf2649d44f36649e534bfd32f3bee597d1fa4bad669f526a5fa015e" exitCode=0
Oct 29 22:10:48.918521 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.918503    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" event={"ID":"4090c710-5f67-4aff-b19c-d02e2347cf77","Type":"ContainerDied","Data":"06bd3eb66bf2649d44f36649e534bfd32f3bee597d1fa4bad669f526a5fa015e"}
Oct 29 22:10:48.918569 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.918538    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9" event={"ID":"4090c710-5f67-4aff-b19c-d02e2347cf77","Type":"ContainerDied","Data":"3fd4614d47e8c8eade32a2f6d2564e973860bc2dd850c699ec7b816fe399c023"}
Oct 29 22:10:48.918629 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.918616    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"
Oct 29 22:10:48.930266 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.930189    4550 scope.go:122] "RemoveContainer" containerID="98453a51df008f8069c330ac05ec7a07e6a634874aef6d40f4084d4a261ba3fc"
Oct 29 22:10:48.933356 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:10:48.933332    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98453a51df008f8069c330ac05ec7a07e6a634874aef6d40f4084d4a261ba3fc\": container with ID starting with 98453a51df008f8069c330ac05ec7a07e6a634874aef6d40f4084d4a261ba3fc not found: ID does not exist" containerID="98453a51df008f8069c330ac05ec7a07e6a634874aef6d40f4084d4a261ba3fc"
Oct 29 22:10:48.933454 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.933367    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98453a51df008f8069c330ac05ec7a07e6a634874aef6d40f4084d4a261ba3fc"} err="failed to get container status \"98453a51df008f8069c330ac05ec7a07e6a634874aef6d40f4084d4a261ba3fc\": rpc error: code = NotFound desc = could not find container \"98453a51df008f8069c330ac05ec7a07e6a634874aef6d40f4084d4a261ba3fc\": container with ID starting with 98453a51df008f8069c330ac05ec7a07e6a634874aef6d40f4084d4a261ba3fc not found: ID does not exist"
Oct 29 22:10:48.933454 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.933395    4550 scope.go:122] "RemoveContainer" containerID="06bd3eb66bf2649d44f36649e534bfd32f3bee597d1fa4bad669f526a5fa015e"
Oct 29 22:10:48.948736 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.948691    4550 scope.go:122] "RemoveContainer" containerID="06bd3eb66bf2649d44f36649e534bfd32f3bee597d1fa4bad669f526a5fa015e"
Oct 29 22:10:48.949137 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:10:48.949111    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06bd3eb66bf2649d44f36649e534bfd32f3bee597d1fa4bad669f526a5fa015e\": container with ID starting with 06bd3eb66bf2649d44f36649e534bfd32f3bee597d1fa4bad669f526a5fa015e not found: ID does not exist" containerID="06bd3eb66bf2649d44f36649e534bfd32f3bee597d1fa4bad669f526a5fa015e"
Oct 29 22:10:48.949137 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.949163    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bd3eb66bf2649d44f36649e534bfd32f3bee597d1fa4bad669f526a5fa015e"} err="failed to get container status \"06bd3eb66bf2649d44f36649e534bfd32f3bee597d1fa4bad669f526a5fa015e\": rpc error: code = NotFound desc = could not find container \"06bd3eb66bf2649d44f36649e534bfd32f3bee597d1fa4bad669f526a5fa015e\": container with ID starting with 06bd3eb66bf2649d44f36649e534bfd32f3bee597d1fa4bad669f526a5fa015e not found: ID does not exist"
Oct 29 22:10:48.995023 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.994988    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-tmp-dir\") pod \"4090c710-5f67-4aff-b19c-d02e2347cf77\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") "
Oct 29 22:10:48.995247 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.995194    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-model-cache\") pod \"4090c710-5f67-4aff-b19c-d02e2347cf77\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") "
Oct 29 22:10:48.995247 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.995224    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-home\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-home\") pod \"4090c710-5f67-4aff-b19c-d02e2347cf77\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") "
Oct 29 22:10:48.995385 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.995341    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e5ef97-3c39-467a-8bf8-60d549360d47-kube-api-access-4nzfw\" (UniqueName: \"kubernetes.io/projected/62e5ef97-3c39-467a-8bf8-60d549360d47-kube-api-access-4nzfw\") pod \"62e5ef97-3c39-467a-8bf8-60d549360d47\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") "
Oct 29 22:10:48.995637 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.995609    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-tmp-dir" pod "4090c710-5f67-4aff-b19c-d02e2347cf77" (UID: "4090c710-5f67-4aff-b19c-d02e2347cf77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:48.995722 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.995632    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-home" pod "4090c710-5f67-4aff-b19c-d02e2347cf77" (UID: "4090c710-5f67-4aff-b19c-d02e2347cf77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:48.995722 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.995655    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-dshm\") pod \"4090c710-5f67-4aff-b19c-d02e2347cf77\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") "
Oct 29 22:10:48.996007 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.995983    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-model-cache" pod "4090c710-5f67-4aff-b19c-d02e2347cf77" (UID: "4090c710-5f67-4aff-b19c-d02e2347cf77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:48.996247 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.996229    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4090c710-5f67-4aff-b19c-d02e2347cf77-tls-certs\" (UniqueName: \"kubernetes.io/secret/4090c710-5f67-4aff-b19c-d02e2347cf77-tls-certs\") pod \"4090c710-5f67-4aff-b19c-d02e2347cf77\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") "
Oct 29 22:10:48.996329 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.996298    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e5ef97-3c39-467a-8bf8-60d549360d47-tls-certs\" (UniqueName: \"kubernetes.io/secret/62e5ef97-3c39-467a-8bf8-60d549360d47-tls-certs\") pod \"62e5ef97-3c39-467a-8bf8-60d549360d47\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") "
Oct 29 22:10:48.996646 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.996619    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-dshm\") pod \"62e5ef97-3c39-467a-8bf8-60d549360d47\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") "
Oct 29 22:10:48.996742 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.996661    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-tmp-dir\") pod \"62e5ef97-3c39-467a-8bf8-60d549360d47\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") "
Oct 29 22:10:48.996742 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.996724    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-home\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-home\") pod \"62e5ef97-3c39-467a-8bf8-60d549360d47\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") "
Oct 29 22:10:48.997322 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.997014    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-tmp-dir" pod "62e5ef97-3c39-467a-8bf8-60d549360d47" (UID: "62e5ef97-3c39-467a-8bf8-60d549360d47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:48.997322 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.997278    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-home" pod "62e5ef97-3c39-467a-8bf8-60d549360d47" (UID: "62e5ef97-3c39-467a-8bf8-60d549360d47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:48.997676 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.997657    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-model-cache\") pod \"62e5ef97-3c39-467a-8bf8-60d549360d47\" (UID: \"62e5ef97-3c39-467a-8bf8-60d549360d47\") "
Oct 29 22:10:48.998086 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.997980    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-model-cache" pod "62e5ef97-3c39-467a-8bf8-60d549360d47" (UID: "62e5ef97-3c39-467a-8bf8-60d549360d47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:48.998280 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.998245    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4090c710-5f67-4aff-b19c-d02e2347cf77-kube-api-access-2v2rn\" (UniqueName: \"kubernetes.io/projected/4090c710-5f67-4aff-b19c-d02e2347cf77-kube-api-access-2v2rn\") pod \"4090c710-5f67-4aff-b19c-d02e2347cf77\" (UID: \"4090c710-5f67-4aff-b19c-d02e2347cf77\") "
Oct 29 22:10:48.998644 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.998617    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-dshm" pod "4090c710-5f67-4aff-b19c-d02e2347cf77" (UID: "4090c710-5f67-4aff-b19c-d02e2347cf77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:48.999309 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.999286    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-dshm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:48.999396 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.999313    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-tmp-dir\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:48.999396 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.999326    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-home\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:48.999396 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.999336    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-model-cache\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:48.999396 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.999350    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-tmp-dir\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:48.999396 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.999364    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-model-cache\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:48.999396 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:48.999375    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4090c710-5f67-4aff-b19c-d02e2347cf77-home\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:49.003808 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.000695    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4090c710-5f67-4aff-b19c-d02e2347cf77-tls-certs" pod "4090c710-5f67-4aff-b19c-d02e2347cf77" (UID: "4090c710-5f67-4aff-b19c-d02e2347cf77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:10:49.003808 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.000864    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-dshm" pod "62e5ef97-3c39-467a-8bf8-60d549360d47" (UID: "62e5ef97-3c39-467a-8bf8-60d549360d47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:49.003808 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.001001    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4090c710-5f67-4aff-b19c-d02e2347cf77-kube-api-access-2v2rn" pod "4090c710-5f67-4aff-b19c-d02e2347cf77" (UID: "4090c710-5f67-4aff-b19c-d02e2347cf77"). InnerVolumeSpecName "kube-api-access-2v2rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:49.003808 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.002373    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e5ef97-3c39-467a-8bf8-60d549360d47-tls-certs" pod "62e5ef97-3c39-467a-8bf8-60d549360d47" (UID: "62e5ef97-3c39-467a-8bf8-60d549360d47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:10:49.012686 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.012659    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e5ef97-3c39-467a-8bf8-60d549360d47-kube-api-access-4nzfw" pod "62e5ef97-3c39-467a-8bf8-60d549360d47" (UID: "62e5ef97-3c39-467a-8bf8-60d549360d47"). InnerVolumeSpecName "kube-api-access-4nzfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:49.102205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.102121    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e5ef97-3c39-467a-8bf8-60d549360d47-tls-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:49.102205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.102202    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e5ef97-3c39-467a-8bf8-60d549360d47-dshm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:49.102416 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.102216    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v2rn\" (UniqueName: \"kubernetes.io/projected/4090c710-5f67-4aff-b19c-d02e2347cf77-kube-api-access-2v2rn\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:49.102416 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.102229    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nzfw\" (UniqueName: \"kubernetes.io/projected/62e5ef97-3c39-467a-8bf8-60d549360d47-kube-api-access-4nzfw\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:49.102416 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.102242    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4090c710-5f67-4aff-b19c-d02e2347cf77-tls-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:10:49.331757 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.331716    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"]
Oct 29 22:10:49.358212 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.342382    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:49.358212 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.349130    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:10:49.358212 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.349237    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm" podUID="8f0d50a6-d0cf-44ae-86e2-32558679df0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 29 22:10:49.358822 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.358610    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/premium-w1-a3c02e-kserve-8f74c5bf5-gz4b9"]
Oct 29 22:10:49.439490 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.439449    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"]
Oct 29 22:10:49.455590 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.445772    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/distinct-w1-a3c02e-kserve-8b8b94c4b-87bnr"]
Oct 29 22:10:49.529307 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:49.524237    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:50.038720 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:50.038676    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" path="/var/lib/kubelet/pods/4090c710-5f67-4aff-b19c-d02e2347cf77/volumes"
Oct 29 22:10:50.039438 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:50.039418    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" path="/var/lib/kubelet/pods/62e5ef97-3c39-467a-8bf8-60d549360d47/volumes"
Oct 29 22:10:50.349543 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:50.349504    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:10:50.349729 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:50.349564    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm" podUID="8f0d50a6-d0cf-44ae-86e2-32558679df0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 29 22:10:50.606251 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:50.602883    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:50.748611 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:50.748575    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:51.153178 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:51.150291    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:51.269044 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:51.269007    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:51.362616 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:51.362574    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:10:51.362616 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:51.362637    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm" podUID="8f0d50a6-d0cf-44ae-86e2-32558679df0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 29 22:10:52.143934 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:52.143902    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:52.359956 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:52.359923    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:52.360506 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:52.360419    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:52.361721 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:52.361696    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:10:52.833900 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:52.833858    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:54.045158 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:54.045105    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:56.120230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:56.120194    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:56.888661 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:56.888618    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:57.002383 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:57.002349    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:58.024576 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:58.024539    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:59.640003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:59.639948    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:59.758382 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:59.758352    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:10:59.885593 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:10:59.885558    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:01.821729 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:01.821684    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:03.535588 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:03.535506    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:05.584758 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:05.584727    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:05.703188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:05.703151    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:07.863571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:07.859241    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:09.539716 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:09.539638    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:10.269355 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:10.269324    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:10.958097 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:10.958043    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:11.091353 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:11.089008    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:11.400538 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:11.400502    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:11.542030 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:11.542001    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:11.578135 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:11:11.578097    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:11:11.983007 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:11.982971    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:12.136225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:12.132759    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:12.445229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:12.445200    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:13.764226 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:13.764190    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:13.889006 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:13.888972    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:15.007095 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:15.007057    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:15.221620 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:15.201780    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:15.803891 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:15.803851    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:15.973720 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:15.973685    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:16.573560 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:16.573519    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:17.298069 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:17.298035    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:17.421020 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:17.420936    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:18.560341 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:18.560311    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:18.687420 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:18.687381    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:19.241419 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:19.241378    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:19.360294 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:19.360226    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:19.913969 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:19.913940    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:20.086102 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:20.080137    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:21.018520 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:21.018481    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:22.616089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:22.616050    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:24.690354 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:24.690323    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:28.164560 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:28.164521    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:28.390903 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:28.390862    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:28.536864 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:28.536784    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:28.926082 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:28.926048    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:30.409640 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:30.409603    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:32.312774 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:32.312246    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:35.317356 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:35.317307    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.318110 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:35.318078    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.383163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:35.382292    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.427344 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:35.427280    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.551167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:35.542470    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:11:35.605166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:35.588924    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:11:35.694324 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:35.694276    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:36.113512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.099670    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:36.236206 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.236160    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:11:36.302164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.295879    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:36.347748 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.341268    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:36.347748 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.341624    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Oct 29 22:11:36.447402 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.402382    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:11:36.471582 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.406010    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:36.483164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.406981    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:36.494129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.432367    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:36.511163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.472855    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:11:36.511474 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.511450    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:11:36.535385 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.535353    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:11:36.558397 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.558327    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:36.596334 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.596292    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:11:36.642321 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.642291    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:11:36.647526 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.647498    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:11:36.763218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.708548    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:36.763218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:36.719439    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:11:37.369839 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:37.369740    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:37.556210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:37.470446    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:37.556210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:37.470929    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:11:37.556210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:37.499128    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:11:37.556210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:37.499971    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:37.556210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:37.503118    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:11:37.556210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:37.509286    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:37.625371 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:37.625268    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:11:37.653288 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:37.653242    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:37.707947 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:37.707896    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:37.708776 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:37.708744    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:37.918582 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:37.918442    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:38.087448 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:38.087383    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:38.444169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:38.427244    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:38.474285 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:38.474233    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:38.558749 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:38.547728    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"]
Oct 29 22:11:38.558749 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:38.547822    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"]
Oct 29 22:11:38.558749 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:38.548414    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f" podUID="e32c602e-d9fc-4a01-93f1-39391f91653d" containerName="istio-proxy" containerID="cri-o://dd69ccf940b46f52d608a13a960546f6ce5b31dd1db7f1b712871f57a20825dd" gracePeriod=30
Oct 29 22:11:38.558749 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:38.549218    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7" podUID="a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" containerName="istio-proxy" containerID="cri-o://ff51d0482e228e94bcf6b6582abd59d7feddd9cbede9ab8a260a17c735ebeef9" gracePeriod=30
Oct 29 22:11:38.622762 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:38.607664    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:38.622762 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:38.615345    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:11:38.682167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:38.643560    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:38.682167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:38.671260    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:11:38.690360 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:38.682623    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:11:38.835233 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:38.743652    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:39.075713 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:39.065520    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"]
Oct 29 22:11:39.075713 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:39.066821    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m" podUID="45d0f73f-8e77-46d0-99e4-9101e92a1f13" containerName="istio-proxy" containerID="cri-o://9edf2c81a4d9c7bbd2d4783bc99591247d91dbe8ff4e68c98f78347b8e76cd0d" gracePeriod=30
Oct 29 22:11:39.084320 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:39.084278    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:11:39.276686 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:39.276585    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:11:39.331450 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:39.329536    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:39.618836 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:39.604281    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:11:39.618836 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:39.607882    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:39.618836 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:39.611471    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:39.633248 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:39.633165    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:11:39.664654 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:39.644756    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:39.688490 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:39.688446    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:11:39.704166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:39.691388    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:39.950220 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:39.926493    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:40.458180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:40.451368    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:40.640173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:40.630465    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:40.640173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:40.630848    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:11:40.640173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:40.631124    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 29 22:11:40.640173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:40.631191    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:11:40.663992 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:40.663947    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:40.683162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:40.664554    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:40.691243 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:40.691202    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:40.768803 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:40.768692    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:40.935328 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:40.935276    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:41.005310 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:41.005261    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" podUID="93b4187a-1551-4cf8-b807-d057baf03920" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:41.564279 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:41.564228    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:41.632759 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:41.632714    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:11:41.633452 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:41.633022    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:11:41.634560 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:41.633057    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:11:41.691859 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:41.691817    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:41.707172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:41.692448    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:11:41.707172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:41.692496    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:11:41.754812 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:11:41.754711    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:11:41.788271 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:41.787311    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:11:41.936897 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:41.936855    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:41.937764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:41.937740    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:11:41.946981 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:41.945505    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:11:41.970313 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:41.970279    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:11:42.122867 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:42.120523    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:42.651361 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:42.651306    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:42.651972 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:42.651935    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:42.673336 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:42.654461    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:11:42.697738 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:42.697696    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:42.820110 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:42.819006    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:11:42.943996 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:42.943410    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:11:42.981745 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:42.980103    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:11:43.237026 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:43.236913    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:43.596595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:43.596479    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:43.633163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:43.630128    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:11:43.670733 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:43.669206    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:43.670733 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:43.669507    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:11:43.679268 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:43.678388    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:43.706674 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:43.705777    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:43.903380 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:43.902066    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:11:44.047215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:44.022472    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:44.656168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:44.641286    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:11:44.656168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:44.641671    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:11:44.672025 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:44.671004    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:44.683227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:44.682244    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:44.727181 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:44.720532    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:44.727710 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:44.727512    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:44.730727 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:44.727920    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:44.760246 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:44.751570    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:11:44.760246 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:44.757218    4550 generic.go:356] "Generic (PLEG): container finished" podID="a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" containerID="ff51d0482e228e94bcf6b6582abd59d7feddd9cbede9ab8a260a17c735ebeef9" exitCode=0
Oct 29 22:11:44.760246 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:44.757370    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7" event={"ID":"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7","Type":"ContainerDied","Data":"ff51d0482e228e94bcf6b6582abd59d7feddd9cbede9ab8a260a17c735ebeef9"}
Oct 29 22:11:44.781163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:44.769553    4550 generic.go:356] "Generic (PLEG): container finished" podID="e32c602e-d9fc-4a01-93f1-39391f91653d" containerID="dd69ccf940b46f52d608a13a960546f6ce5b31dd1db7f1b712871f57a20825dd" exitCode=0
Oct 29 22:11:44.781163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:44.769595    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f" event={"ID":"e32c602e-d9fc-4a01-93f1-39391f91653d","Type":"ContainerDied","Data":"dd69ccf940b46f52d608a13a960546f6ce5b31dd1db7f1b712871f57a20825dd"}
Oct 29 22:11:44.923530 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:44.922610    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:45.145783 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.145737    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:11:45.216693 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.216504    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:11:45.336455 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.336354    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:11:45.383724 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.383692    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 22:11:45.393746 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.392748    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-token\" (UniqueName: \"kubernetes.io/projected/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-token\") pod \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") "
Oct 29 22:11:45.398014 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.397936    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-token" pod "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" (UID: "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:45.413601 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.413567    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"]
Oct 29 22:11:45.414728 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.414706    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main"
Oct 29 22:11:45.414913 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.414859    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main"
Oct 29 22:11:45.415046 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.415034    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e32c602e-d9fc-4a01-93f1-39391f91653d" containerName="istio-proxy"
Oct 29 22:11:45.415122 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.415112    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32c602e-d9fc-4a01-93f1-39391f91653d" containerName="istio-proxy"
Oct 29 22:11:45.415244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.415227    4550 cpu_manager.go:414] "containerMap: removing container" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main"
Oct 29 22:11:45.415244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.415242    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main"
Oct 29 22:11:45.415362 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.415261    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" containerName="istio-proxy"
Oct 29 22:11:45.415362 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.415269    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" containerName="istio-proxy"
Oct 29 22:11:45.415478 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.415465    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32c602e-d9fc-4a01-93f1-39391f91653d" containerName="istio-proxy"
Oct 29 22:11:45.415534 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.415484    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e5ef97-3c39-467a-8bf8-60d549360d47" containerName="main"
Oct 29 22:11:45.415534 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.415494    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" containerName="istio-proxy"
Oct 29 22:11:45.415534 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.415515    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4090c710-5f67-4aff-b19c-d02e2347cf77" containerName="main"
Oct 29 22:11:45.419635 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.419611    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.434504 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.434475    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-75db5b-openshift-default-dockercfg-mwm8c\""
Oct 29 22:11:45.439855 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.439811    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"]
Oct 29 22:11:45.444340 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.444315    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 22:11:45.494877 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.494847    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-podinfo\") pod \"e32c602e-d9fc-4a01-93f1-39391f91653d\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") "
Oct 29 22:11:45.495049 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.494942    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-data\") pod \"e32c602e-d9fc-4a01-93f1-39391f91653d\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") "
Oct 29 22:11:45.495049 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.494986    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-workload-certs\") pod \"e32c602e-d9fc-4a01-93f1-39391f91653d\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") "
Oct 29 22:11:45.495049 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495026    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-credential-socket\") pod \"e32c602e-d9fc-4a01-93f1-39391f91653d\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") "
Oct 29 22:11:45.495244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495055    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-token\" (UniqueName: \"kubernetes.io/projected/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-token\") pod \"e32c602e-d9fc-4a01-93f1-39391f91653d\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") "
Oct 29 22:11:45.495244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495154    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-workload-socket\") pod \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") "
Oct 29 22:11:45.495244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495199    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-workload-socket\") pod \"e32c602e-d9fc-4a01-93f1-39391f91653d\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") "
Oct 29 22:11:45.495397 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495278    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32c602e-d9fc-4a01-93f1-39391f91653d-kube-api-access-hpk87\" (UniqueName: \"kubernetes.io/projected/e32c602e-d9fc-4a01-93f1-39391f91653d-kube-api-access-hpk87\") pod \"e32c602e-d9fc-4a01-93f1-39391f91653d\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") "
Oct 29 22:11:45.495397 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495300    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-data" pod "e32c602e-d9fc-4a01-93f1-39391f91653d" (UID: "e32c602e-d9fc-4a01-93f1-39391f91653d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:45.495397 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495360    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-credential-socket\") pod \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") "
Oct 29 22:11:45.495558 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495530    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istiod-ca-cert\") pod \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") "
Oct 29 22:11:45.495619 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495575    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-envoy\") pod \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") "
Oct 29 22:11:45.495619 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495556    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-credential-socket" pod "e32c602e-d9fc-4a01-93f1-39391f91653d" (UID: "e32c602e-d9fc-4a01-93f1-39391f91653d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:45.495727 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495626    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-podinfo\") pod \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") "
Oct 29 22:11:45.495727 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495656    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-data\") pod \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") "
Oct 29 22:11:45.495727 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495657    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-credential-socket" pod "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" (UID: "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:45.495727 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495684    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-envoy\") pod \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") "
Oct 29 22:11:45.495727 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495724    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-kube-api-access-tchb9\" (UniqueName: \"kubernetes.io/projected/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-kube-api-access-tchb9\") pod \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") "
Oct 29 22:11:45.496000 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495752    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-workload-socket" pod "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" (UID: "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:45.496000 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495753    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-credential-socket\") pod \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") "
Oct 29 22:11:45.496000 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495818    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-workload-certs\") pod \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") "
Oct 29 22:11:45.496000 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495875    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-envoy\") pod \"e32c602e-d9fc-4a01-93f1-39391f91653d\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") "
Oct 29 22:11:45.496000 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495993    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-credential-socket" pod "45d0f73f-8e77-46d0-99e4-9101e92a1f13" (UID: "45d0f73f-8e77-46d0-99e4-9101e92a1f13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:45.496287 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.495999    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-workload-socket" pod "e32c602e-d9fc-4a01-93f1-39391f91653d" (UID: "e32c602e-d9fc-4a01-93f1-39391f91653d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:45.496287 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.496064    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-workload-socket\") pod \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") "
Oct 29 22:11:45.496287 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.496104    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e32c602e-d9fc-4a01-93f1-39391f91653d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32c602e-d9fc-4a01-93f1-39391f91653d-istiod-ca-cert\") pod \"e32c602e-d9fc-4a01-93f1-39391f91653d\" (UID: \"e32c602e-d9fc-4a01-93f1-39391f91653d\") "
Oct 29 22:11:45.496287 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.496231    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-data\") pod \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\" (UID: \"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7\") "
Oct 29 22:11:45.496494 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.496367    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istiod-ca-cert" pod "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" (UID: "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:45.496494 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.496438    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-workload-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.496494 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.496485    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-credential-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.496687 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.496595    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-token\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.496687 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.496657    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b27b23e3-07bd-4531-9d64-73d1b66de485-istiod-ca-cert\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.496797 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.496770    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-workload-certs" pod "e32c602e-d9fc-4a01-93f1-39391f91653d" (UID: "e32c602e-d9fc-4a01-93f1-39391f91653d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:45.496862 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.496794    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-envoy\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.497026 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497003    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-data\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.497100 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497024    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-workload-certs" pod "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" (UID: "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:45.497210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497135    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsm75\" (UniqueName: \"kubernetes.io/projected/b27b23e3-07bd-4531-9d64-73d1b66de485-kube-api-access-hsm75\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.497288 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497257    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-workload-certs\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.497288 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497270    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-workload-socket" pod "45d0f73f-8e77-46d0-99e4-9101e92a1f13" (UID: "45d0f73f-8e77-46d0-99e4-9101e92a1f13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:45.497394 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497362    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-podinfo\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.497501 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497479    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.497571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497505    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.497571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497519    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.497571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497532    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.497571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497546    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.497571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497560    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.497845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497572    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.497845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497587    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.497845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497600    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.497845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497613    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.497845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.497626    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.498096 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.498017    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32c602e-d9fc-4a01-93f1-39391f91653d-istiod-ca-cert" pod "e32c602e-d9fc-4a01-93f1-39391f91653d" (UID: "e32c602e-d9fc-4a01-93f1-39391f91653d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:45.498321 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.498290    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-data" pod "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" (UID: "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:45.498603 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.498580    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-data" pod "45d0f73f-8e77-46d0-99e4-9101e92a1f13" (UID: "45d0f73f-8e77-46d0-99e4-9101e92a1f13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:45.499047 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.499007    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-envoy" pod "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" (UID: "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:45.500670 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.500644    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-token" pod "e32c602e-d9fc-4a01-93f1-39391f91653d" (UID: "e32c602e-d9fc-4a01-93f1-39391f91653d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:45.500805 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.500784    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-podinfo" pod "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" (UID: "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:11:45.500847 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.500796    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32c602e-d9fc-4a01-93f1-39391f91653d-kube-api-access-hpk87" pod "e32c602e-d9fc-4a01-93f1-39391f91653d" (UID: "e32c602e-d9fc-4a01-93f1-39391f91653d"). InnerVolumeSpecName "kube-api-access-hpk87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:45.501090 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.501062    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-envoy" pod "45d0f73f-8e77-46d0-99e4-9101e92a1f13" (UID: "45d0f73f-8e77-46d0-99e4-9101e92a1f13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:45.501217 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.501171    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-podinfo" pod "e32c602e-d9fc-4a01-93f1-39391f91653d" (UID: "e32c602e-d9fc-4a01-93f1-39391f91653d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:11:45.501831 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.501810    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-envoy" pod "e32c602e-d9fc-4a01-93f1-39391f91653d" (UID: "e32c602e-d9fc-4a01-93f1-39391f91653d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:45.502124 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.502105    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-kube-api-access-tchb9" pod "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" (UID: "a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7"). InnerVolumeSpecName "kube-api-access-tchb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:45.598352 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.598320    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d0f73f-8e77-46d0-99e4-9101e92a1f13-kube-api-access-9jlqk\" (UniqueName: \"kubernetes.io/projected/45d0f73f-8e77-46d0-99e4-9101e92a1f13-kube-api-access-9jlqk\") pod \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") "
Oct 29 22:11:45.598519 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.598491    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istiod-ca-cert\") pod \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") "
Oct 29 22:11:45.598575 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.598553    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-token\" (UniqueName: \"kubernetes.io/projected/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-token\") pod \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") "
Oct 29 22:11:45.598575 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.598570    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-podinfo\") pod \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") "
Oct 29 22:11:45.598690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.598622    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-workload-certs\") pod \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\" (UID: \"45d0f73f-8e77-46d0-99e4-9101e92a1f13\") "
Oct 29 22:11:45.598886 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.598859    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsm75\" (UniqueName: \"kubernetes.io/projected/b27b23e3-07bd-4531-9d64-73d1b66de485-kube-api-access-hsm75\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.599072 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.598912    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istiod-ca-cert" pod "45d0f73f-8e77-46d0-99e4-9101e92a1f13" (UID: "45d0f73f-8e77-46d0-99e4-9101e92a1f13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:45.599072 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.598920    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-workload-certs\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.599072 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.598966    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-workload-certs" pod "45d0f73f-8e77-46d0-99e4-9101e92a1f13" (UID: "45d0f73f-8e77-46d0-99e4-9101e92a1f13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:45.599072 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.599007    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-podinfo\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.599072 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.599056    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-workload-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.599701 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.599094    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-credential-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.599701 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.599229    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-token\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.599701 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.599285    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b27b23e3-07bd-4531-9d64-73d1b66de485-istiod-ca-cert\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.599701 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.599299    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-workload-certs\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.599701 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.599690    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-credential-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.599974 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.599733    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-envoy\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.599974 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.599826    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-workload-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.599974 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.599960    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-data\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.600178 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.600091    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b27b23e3-07bd-4531-9d64-73d1b66de485-istiod-ca-cert\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.600178 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.600156    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.600178 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.600176    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.600346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.600189    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.600346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.600202    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.600346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.600214    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.600346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.600228    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpk87\" (UniqueName: \"kubernetes.io/projected/e32c602e-d9fc-4a01-93f1-39391f91653d-kube-api-access-hpk87\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.600346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.600241    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.600346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.600256    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.600346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.600269    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.600346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.600281    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.600346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.600295    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tchb9\" (UniqueName: \"kubernetes.io/projected/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7-kube-api-access-tchb9\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.600346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.600310    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32c602e-d9fc-4a01-93f1-39391f91653d-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.600346 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.600326    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32c602e-d9fc-4a01-93f1-39391f91653d-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.600958 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.600375    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-data\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.601044 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.601020    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d0f73f-8e77-46d0-99e4-9101e92a1f13-kube-api-access-9jlqk" pod "45d0f73f-8e77-46d0-99e4-9101e92a1f13" (UID: "45d0f73f-8e77-46d0-99e4-9101e92a1f13"). InnerVolumeSpecName "kube-api-access-9jlqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:45.601125 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.601102    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-podinfo" pod "45d0f73f-8e77-46d0-99e4-9101e92a1f13" (UID: "45d0f73f-8e77-46d0-99e4-9101e92a1f13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:11:45.601234 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.601119    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-token" pod "45d0f73f-8e77-46d0-99e4-9101e92a1f13" (UID: "45d0f73f-8e77-46d0-99e4-9101e92a1f13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:45.601804 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.601782    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-podinfo\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.601978 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.601962    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-envoy\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.621848 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.621822    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-token\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.621973 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.621957    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsm75\" (UniqueName: \"kubernetes.io/projected/b27b23e3-07bd-4531-9d64-73d1b66de485-kube-api-access-hsm75\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.701650 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.701579    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.701650 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.701606    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d0f73f-8e77-46d0-99e4-9101e92a1f13-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.701650 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.701615    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jlqk\" (UniqueName: \"kubernetes.io/projected/45d0f73f-8e77-46d0-99e4-9101e92a1f13-kube-api-access-9jlqk\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:45.766541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.766516    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:45.778673 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.778647    4550 generic.go:356] "Generic (PLEG): container finished" podID="45d0f73f-8e77-46d0-99e4-9101e92a1f13" containerID="9edf2c81a4d9c7bbd2d4783bc99591247d91dbe8ff4e68c98f78347b8e76cd0d" exitCode=0
Oct 29 22:11:45.778787 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.778691    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"
Oct 29 22:11:45.778787 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.778725    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m" event={"ID":"45d0f73f-8e77-46d0-99e4-9101e92a1f13","Type":"ContainerDied","Data":"9edf2c81a4d9c7bbd2d4783bc99591247d91dbe8ff4e68c98f78347b8e76cd0d"}
Oct 29 22:11:45.778787 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.778782    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m" event={"ID":"45d0f73f-8e77-46d0-99e4-9101e92a1f13","Type":"ContainerDied","Data":"0e8968a0b5919756b52570cbcf16fcbd1884b671f9c254fa00a89e9584830267"}
Oct 29 22:11:45.778961 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.778811    4550 scope.go:122] "RemoveContainer" containerID="9edf2c81a4d9c7bbd2d4783bc99591247d91dbe8ff4e68c98f78347b8e76cd0d"
Oct 29 22:11:45.780718 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.780699    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"
Oct 29 22:11:45.780843 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.780710    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7" event={"ID":"a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7","Type":"ContainerDied","Data":"72d2811493300928f5b9dec97841967ef972876cf1889117295feb9f67916921"}
Oct 29 22:11:45.782699 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.782588    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f" event={"ID":"e32c602e-d9fc-4a01-93f1-39391f91653d","Type":"ContainerDied","Data":"78c1a8f137a70455b96f426e7cb160835961e052adcab0a286c4d7b58aa9c8eb"}
Oct 29 22:11:45.782699 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.782684    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"
Oct 29 22:11:45.791991 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.791972    4550 scope.go:122] "RemoveContainer" containerID="9edf2c81a4d9c7bbd2d4783bc99591247d91dbe8ff4e68c98f78347b8e76cd0d"
Oct 29 22:11:45.792306 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:11:45.792286    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9edf2c81a4d9c7bbd2d4783bc99591247d91dbe8ff4e68c98f78347b8e76cd0d\": container with ID starting with 9edf2c81a4d9c7bbd2d4783bc99591247d91dbe8ff4e68c98f78347b8e76cd0d not found: ID does not exist" containerID="9edf2c81a4d9c7bbd2d4783bc99591247d91dbe8ff4e68c98f78347b8e76cd0d"
Oct 29 22:11:45.792391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.792322    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9edf2c81a4d9c7bbd2d4783bc99591247d91dbe8ff4e68c98f78347b8e76cd0d"} err="failed to get container status \"9edf2c81a4d9c7bbd2d4783bc99591247d91dbe8ff4e68c98f78347b8e76cd0d\": rpc error: code = NotFound desc = could not find container \"9edf2c81a4d9c7bbd2d4783bc99591247d91dbe8ff4e68c98f78347b8e76cd0d\": container with ID starting with 9edf2c81a4d9c7bbd2d4783bc99591247d91dbe8ff4e68c98f78347b8e76cd0d not found: ID does not exist"
Oct 29 22:11:45.792391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.792339    4550 scope.go:122] "RemoveContainer" containerID="ff51d0482e228e94bcf6b6582abd59d7feddd9cbede9ab8a260a17c735ebeef9"
Oct 29 22:11:45.804222 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.804203    4550 scope.go:122] "RemoveContainer" containerID="dd69ccf940b46f52d608a13a960546f6ce5b31dd1db7f1b712871f57a20825dd"
Oct 29 22:11:45.868129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.868097    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"]
Oct 29 22:11:45.874185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.874156    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-slv2m"]
Oct 29 22:11:45.903698 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.903662    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"]
Oct 29 22:11:45.908735 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.908700    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-85qf7"]
Oct 29 22:11:45.944041 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.944007    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"]
Oct 29 22:11:45.948058 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.948029    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-8n89f"]
Oct 29 22:11:45.990765 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.990696    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45d0f73f-8e77-46d0-99e4-9101e92a1f13" path="/var/lib/kubelet/pods/45d0f73f-8e77-46d0-99e4-9101e92a1f13/volumes"
Oct 29 22:11:45.991475 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.991451    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7" path="/var/lib/kubelet/pods/a3c9a8e5-9324-4ded-bdb1-03e9e9ff7da7/volumes"
Oct 29 22:11:45.992072 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:45.992051    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32c602e-d9fc-4a01-93f1-39391f91653d" path="/var/lib/kubelet/pods/e32c602e-d9fc-4a01-93f1-39391f91653d/volumes"
Oct 29 22:11:46.098134 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:46.098105    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"]
Oct 29 22:11:46.098134 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:11:46.098122    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27b23e3_07bd_4531_9d64_73d1b66de485.slice/crio-59e3f45e3d8083afa4aad7fcee60e3d447a4370dcb7d8402d66bc1239e787a89 WatchSource:0}: Error finding container 59e3f45e3d8083afa4aad7fcee60e3d447a4370dcb7d8402d66bc1239e787a89: Status 404 returned error can't find the container with id 59e3f45e3d8083afa4aad7fcee60e3d447a4370dcb7d8402d66bc1239e787a89
Oct 29 22:11:46.101382 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:46.101348    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:11:46.101512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:46.101429    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:11:46.101512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:46.101485    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:11:46.662840 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:46.662805    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:46.791315 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:46.791275    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p" event={"ID":"b27b23e3-07bd-4531-9d64-73d1b66de485","Type":"ContainerStarted","Data":"3b1a1e2579e8bf8fe938ff45559dae6bb7a27cd5f164817a07c4da911fa170cf"}
Oct 29 22:11:46.791754 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:46.791378    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p" event={"ID":"b27b23e3-07bd-4531-9d64-73d1b66de485","Type":"ContainerStarted","Data":"59e3f45e3d8083afa4aad7fcee60e3d447a4370dcb7d8402d66bc1239e787a89"}
Oct 29 22:11:46.840201 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:46.840116    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p" podStartSLOduration=7.840101595 podStartE2EDuration="7.840101595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:46.837474426 +0000 UTC m=+1522.994599940" watchObservedRunningTime="2026-10-29 22:11:46.840101595 +0000 UTC m=+1522.997227092"
Oct 29 22:11:47.767489 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:47.767456    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:47.774780 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:47.774754    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:47.801704 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:47.801675    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:47.802537 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:47.802517    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:11:48.713848 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:48.713812    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:49.353385 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:49.353355    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:49.559791 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:49.559758    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:51.173999 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:51.173959    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:51.586559 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:51.586525    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:51.728077 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:51.728029    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:51.838217 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:51.838090    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"]
Oct 29 22:11:51.863154 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:51.839113    4550 cpu_manager.go:414] "containerMap: removing container" podUID="45d0f73f-8e77-46d0-99e4-9101e92a1f13" containerName="istio-proxy"
Oct 29 22:11:51.863154 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:51.839134    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d0f73f-8e77-46d0-99e4-9101e92a1f13" containerName="istio-proxy"
Oct 29 22:11:51.863154 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:51.839329    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d0f73f-8e77-46d0-99e4-9101e92a1f13" containerName="istio-proxy"
Oct 29 22:11:51.863154 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:51.852398    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"
Oct 29 22:11:51.882204 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:51.880749    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-75db5b\""
Oct 29 22:11:51.888205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:51.884888    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"]
Oct 29 22:11:51.919504 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:51.919456    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8020c6d3-1330-4581-8824-2ca1bdf0bc35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5\" (UID: \"8020c6d3-1330-4581-8824-2ca1bdf0bc35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"
Oct 29 22:11:51.919745 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:51.919719    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkhgv\" (UniqueName: \"kubernetes.io/projected/8020c6d3-1330-4581-8824-2ca1bdf0bc35-kube-api-access-tkhgv\") pod \"payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5\" (UID: \"8020c6d3-1330-4581-8824-2ca1bdf0bc35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"
Oct 29 22:11:51.989166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:51.988510    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-75db5b-dockercfg-pssln\""
Oct 29 22:11:52.030156 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:52.023216    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8020c6d3-1330-4581-8824-2ca1bdf0bc35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5\" (UID: \"8020c6d3-1330-4581-8824-2ca1bdf0bc35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"
Oct 29 22:11:52.030156 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:52.023305    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkhgv\" (UniqueName: \"kubernetes.io/projected/8020c6d3-1330-4581-8824-2ca1bdf0bc35-kube-api-access-tkhgv\") pod \"payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5\" (UID: \"8020c6d3-1330-4581-8824-2ca1bdf0bc35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"
Oct 29 22:11:52.030156 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:52.028799    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8020c6d3-1330-4581-8824-2ca1bdf0bc35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5\" (UID: \"8020c6d3-1330-4581-8824-2ca1bdf0bc35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"
Oct 29 22:11:52.059369 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:52.059322    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkhgv\" (UniqueName: \"kubernetes.io/projected/8020c6d3-1330-4581-8824-2ca1bdf0bc35-kube-api-access-tkhgv\") pod \"payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5\" (UID: \"8020c6d3-1330-4581-8824-2ca1bdf0bc35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"
Oct 29 22:11:52.238666 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:52.238566    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"
Oct 29 22:11:52.552617 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:52.552588    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"]
Oct 29 22:11:52.554207 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:11:52.554181    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8020c6d3_1330_4581_8824_2ca1bdf0bc35.slice/crio-ae0047ebebbf60abc4d31a62dfd87846afaefdfcf56c2feaddc741ec437987ec WatchSource:0}: Error finding container ae0047ebebbf60abc4d31a62dfd87846afaefdfcf56c2feaddc741ec437987ec: Status 404 returned error can't find the container with id ae0047ebebbf60abc4d31a62dfd87846afaefdfcf56c2feaddc741ec437987ec
Oct 29 22:11:52.885433 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:52.885392    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5" event={"ID":"8020c6d3-1330-4581-8824-2ca1bdf0bc35","Type":"ContainerStarted","Data":"057e534b1b2915fa8ccd7cf5c28a62bfc635a63797b948c2ab62d352286c8996"}
Oct 29 22:11:52.885633 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:52.885449    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5" event={"ID":"8020c6d3-1330-4581-8824-2ca1bdf0bc35","Type":"ContainerStarted","Data":"ae0047ebebbf60abc4d31a62dfd87846afaefdfcf56c2feaddc741ec437987ec"}
Oct 29 22:11:52.885633 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:52.885469    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"
Oct 29 22:11:52.889899 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:52.889856    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:52.932918 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:52.931786    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5" podStartSLOduration=1.931765988 podStartE2EDuration="1.931765988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:52.928646441 +0000 UTC m=+1529.085771936" watchObservedRunningTime="2026-10-29 22:11:52.931765988 +0000 UTC m=+1529.088891491"
Oct 29 22:11:53.079168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:53.079072    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:53.215298 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:53.214639    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"]
Oct 29 22:11:53.215298 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:53.214910    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerName="istio-proxy" containerID="cri-o://909c1cfa9b74bf779dad64d01d06fc589c01d337c386a8526095d7d53fd69fca" gracePeriod=30
Oct 29 22:11:53.221210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:53.221185    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"]
Oct 29 22:11:53.228359 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:53.228328    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj" podUID="e8bb2593-5acc-40c9-9025-e5e6b29bd80b" containerName="istio-proxy" containerID="cri-o://f6313501eac3e27eb555b6e5c311f8385406512bb62587cddfdbb177af894661" gracePeriod=30
Oct 29 22:11:53.235115 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:53.235093    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"]
Oct 29 22:11:53.235686 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:53.235627    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp" podUID="7fe0d723-a137-4c93-9657-ca9258b08a27" containerName="istio-proxy" containerID="cri-o://34d3593ee7810d93590353aefc91ba1dad61c15c9a3a7b49ec8514e771475898" gracePeriod=30
Oct 29 22:11:53.240772 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:53.240692    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:53.525627 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:53.525465    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:53.713807 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:53.713768    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:53.836705 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:53.836663    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:55.351995 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:55.351959    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:56.616245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:56.616193    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:11:56.616654 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:56.616276    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp" podUID="7fe0d723-a137-4c93-9657-ca9258b08a27" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:11:57.817549 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:57.817520    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:58.551411 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.551379    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 22:11:58.606032 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.606003    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-envoy\") pod \"7fe0d723-a137-4c93-9657-ca9258b08a27\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") "
Oct 29 22:11:58.606219 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.606197    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-workload-certs\") pod \"7fe0d723-a137-4c93-9657-ca9258b08a27\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") "
Oct 29 22:11:58.606312 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.606265    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-data\") pod \"7fe0d723-a137-4c93-9657-ca9258b08a27\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") "
Oct 29 22:11:58.606378 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.606308    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-credential-socket\") pod \"7fe0d723-a137-4c93-9657-ca9258b08a27\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") "
Oct 29 22:11:58.606378 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.606335    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-workload-socket\") pod \"7fe0d723-a137-4c93-9657-ca9258b08a27\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") "
Oct 29 22:11:58.606378 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.606356    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe0d723-a137-4c93-9657-ca9258b08a27-kube-api-access-878dg\" (UniqueName: \"kubernetes.io/projected/7fe0d723-a137-4c93-9657-ca9258b08a27-kube-api-access-878dg\") pod \"7fe0d723-a137-4c93-9657-ca9258b08a27\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") "
Oct 29 22:11:58.606538 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.606410    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-podinfo\") pod \"7fe0d723-a137-4c93-9657-ca9258b08a27\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") "
Oct 29 22:11:58.606538 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.606446    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-workload-certs" pod "7fe0d723-a137-4c93-9657-ca9258b08a27" (UID: "7fe0d723-a137-4c93-9657-ca9258b08a27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:58.606538 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.606463    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe0d723-a137-4c93-9657-ca9258b08a27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe0d723-a137-4c93-9657-ca9258b08a27-istiod-ca-cert\") pod \"7fe0d723-a137-4c93-9657-ca9258b08a27\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") "
Oct 29 22:11:58.606703 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.606599    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-token\" (UniqueName: \"kubernetes.io/projected/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-token\") pod \"7fe0d723-a137-4c93-9657-ca9258b08a27\" (UID: \"7fe0d723-a137-4c93-9657-ca9258b08a27\") "
Oct 29 22:11:58.606784 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.606744    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-workload-socket" pod "7fe0d723-a137-4c93-9657-ca9258b08a27" (UID: "7fe0d723-a137-4c93-9657-ca9258b08a27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:58.606991 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.606973    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-data" pod "7fe0d723-a137-4c93-9657-ca9258b08a27" (UID: "7fe0d723-a137-4c93-9657-ca9258b08a27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:58.607491 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.607469    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.607638 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.607495    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.607638 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.607508    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.607793 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.607722    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-credential-socket" pod "7fe0d723-a137-4c93-9657-ca9258b08a27" (UID: "7fe0d723-a137-4c93-9657-ca9258b08a27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:58.608362 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.608337    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe0d723-a137-4c93-9657-ca9258b08a27-istiod-ca-cert" pod "7fe0d723-a137-4c93-9657-ca9258b08a27" (UID: "7fe0d723-a137-4c93-9657-ca9258b08a27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:58.608857 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.608828    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-envoy" pod "7fe0d723-a137-4c93-9657-ca9258b08a27" (UID: "7fe0d723-a137-4c93-9657-ca9258b08a27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:58.609009 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.608985    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-podinfo" pod "7fe0d723-a137-4c93-9657-ca9258b08a27" (UID: "7fe0d723-a137-4c93-9657-ca9258b08a27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:11:58.609108 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.609069    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-token" pod "7fe0d723-a137-4c93-9657-ca9258b08a27" (UID: "7fe0d723-a137-4c93-9657-ca9258b08a27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:58.610084 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.610066    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:11:58.610187 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.610162    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe0d723-a137-4c93-9657-ca9258b08a27-kube-api-access-878dg" pod "7fe0d723-a137-4c93-9657-ca9258b08a27" (UID: "7fe0d723-a137-4c93-9657-ca9258b08a27"). InnerVolumeSpecName "kube-api-access-878dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:58.618799 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.618755    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:11:58.708471 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.708436    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-workload-socket\") pod \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") "
Oct 29 22:11:58.708916 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.708855    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-workload-socket" pod "e8bb2593-5acc-40c9-9025-e5e6b29bd80b" (UID: "e8bb2593-5acc-40c9-9025-e5e6b29bd80b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:58.709884 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.709852    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-kube-api-access-cbnxh\" (UniqueName: \"kubernetes.io/projected/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-kube-api-access-cbnxh\") pod \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") "
Oct 29 22:11:58.709999 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.709983    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-credential-socket\") pod \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") "
Oct 29 22:11:58.710111 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710092    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-token\" (UniqueName: \"kubernetes.io/projected/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-token\") pod \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") "
Oct 29 22:11:58.710217 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710199    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istiod-ca-cert\") pod \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") "
Oct 29 22:11:58.710282 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710245    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-envoy\") pod \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") "
Oct 29 22:11:58.710338 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710301    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-credential-socket\") pod \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") "
Oct 29 22:11:58.710404 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710339    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-podinfo\") pod \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") "
Oct 29 22:11:58.710404 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710399    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istiod-ca-cert\") pod \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") "
Oct 29 22:11:58.710551 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710427    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-credential-socket" pod "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" (UID: "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:58.710551 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710451    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-envoy\") pod \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") "
Oct 29 22:11:58.710551 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710520    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-workload-certs\") pod \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") "
Oct 29 22:11:58.710760 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710551    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-data\") pod \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") "
Oct 29 22:11:58.710760 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710599    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-token\" (UniqueName: \"kubernetes.io/projected/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-token\") pod \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") "
Oct 29 22:11:58.710760 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710671    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-workload-certs\") pod \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") "
Oct 29 22:11:58.710760 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710720    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-kube-api-access-mphpn\" (UniqueName: \"kubernetes.io/projected/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-kube-api-access-mphpn\") pod \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") "
Oct 29 22:11:58.710975 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710776    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-data\") pod \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") "
Oct 29 22:11:58.710975 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710810    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-podinfo\") pod \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\" (UID: \"e8bb2593-5acc-40c9-9025-e5e6b29bd80b\") "
Oct 29 22:11:58.710975 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.710888    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-workload-socket\") pod \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\" (UID: \"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb\") "
Oct 29 22:11:58.711820 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.711798    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.711896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.711825    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.711896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.711828    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-data" pod "e8bb2593-5acc-40c9-9025-e5e6b29bd80b" (UID: "e8bb2593-5acc-40c9-9025-e5e6b29bd80b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:58.711896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.711840    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe0d723-a137-4c93-9657-ca9258b08a27-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.711896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.711855    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-878dg\" (UniqueName: \"kubernetes.io/projected/7fe0d723-a137-4c93-9657-ca9258b08a27-kube-api-access-878dg\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.711896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.711868    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.711896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.711882    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.712118 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.711896    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe0d723-a137-4c93-9657-ca9258b08a27-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.712118 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.711909    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe0d723-a137-4c93-9657-ca9258b08a27-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.712118 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.712029    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-credential-socket" pod "e8bb2593-5acc-40c9-9025-e5e6b29bd80b" (UID: "e8bb2593-5acc-40c9-9025-e5e6b29bd80b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:58.713167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.713015    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-data" pod "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" (UID: "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:58.713167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.713114    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-kube-api-access-cbnxh" pod "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" (UID: "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb"). InnerVolumeSpecName "kube-api-access-cbnxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:58.713426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.713282    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-workload-certs" pod "e8bb2593-5acc-40c9-9025-e5e6b29bd80b" (UID: "e8bb2593-5acc-40c9-9025-e5e6b29bd80b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:58.713426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.713374    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-workload-certs" pod "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" (UID: "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:58.714032 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.713991    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-token" pod "e8bb2593-5acc-40c9-9025-e5e6b29bd80b" (UID: "e8bb2593-5acc-40c9-9025-e5e6b29bd80b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:58.716431 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.716406    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-envoy" pod "e8bb2593-5acc-40c9-9025-e5e6b29bd80b" (UID: "e8bb2593-5acc-40c9-9025-e5e6b29bd80b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:58.719492 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.719077    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istiod-ca-cert" pod "e8bb2593-5acc-40c9-9025-e5e6b29bd80b" (UID: "e8bb2593-5acc-40c9-9025-e5e6b29bd80b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:58.721427 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.720906    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istiod-ca-cert" pod "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" (UID: "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:58.721427 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.721378    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-workload-socket" pod "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" (UID: "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:58.724512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.724474    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-podinfo" pod "e8bb2593-5acc-40c9-9025-e5e6b29bd80b" (UID: "e8bb2593-5acc-40c9-9025-e5e6b29bd80b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:11:58.724606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.724516    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-podinfo" pod "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" (UID: "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:11:58.724836 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.724811    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-envoy" pod "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" (UID: "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:58.725575 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.725544    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-token" pod "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" (UID: "f1fefa99-0a5c-422e-b950-89e0a7f2c1cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:58.726900 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.726870    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-kube-api-access-mphpn" pod "e8bb2593-5acc-40c9-9025-e5e6b29bd80b" (UID: "e8bb2593-5acc-40c9-9025-e5e6b29bd80b"). InnerVolumeSpecName "kube-api-access-mphpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:58.814106 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814032    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.814106 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814060    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbnxh\" (UniqueName: \"kubernetes.io/projected/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-kube-api-access-cbnxh\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.814106 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814071    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.814106 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814080    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.814106 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814088    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.814106 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814097    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.814106 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814107    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.814442 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814115    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.814442 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814123    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.814442 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814132    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.814442 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814152    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.814442 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814160    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.814442 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814167    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.814442 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814176    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mphpn\" (UniqueName: \"kubernetes.io/projected/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-kube-api-access-mphpn\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.814442 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814184    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.814442 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.814192    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bb2593-5acc-40c9-9025-e5e6b29bd80b-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.956243 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.947060    4550 generic.go:356] "Generic (PLEG): container finished" podID="7fe0d723-a137-4c93-9657-ca9258b08a27" containerID="34d3593ee7810d93590353aefc91ba1dad61c15c9a3a7b49ec8514e771475898" exitCode=0
Oct 29 22:11:58.956243 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.947285    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp" event={"ID":"7fe0d723-a137-4c93-9657-ca9258b08a27","Type":"ContainerDied","Data":"34d3593ee7810d93590353aefc91ba1dad61c15c9a3a7b49ec8514e771475898"}
Oct 29 22:11:58.956243 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.947347    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp" event={"ID":"7fe0d723-a137-4c93-9657-ca9258b08a27","Type":"ContainerDied","Data":"d49e3aadecfb96a2d1ef7029ed419f05f7ed9e821b1e92c0d199d08850facac7"}
Oct 29 22:11:58.956243 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.947381    4550 scope.go:122] "RemoveContainer" containerID="34d3593ee7810d93590353aefc91ba1dad61c15c9a3a7b49ec8514e771475898"
Oct 29 22:11:58.956243 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.947515    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"
Oct 29 22:11:58.963635 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.963593    4550 generic.go:356] "Generic (PLEG): container finished" podID="e8bb2593-5acc-40c9-9025-e5e6b29bd80b" containerID="f6313501eac3e27eb555b6e5c311f8385406512bb62587cddfdbb177af894661" exitCode=0
Oct 29 22:11:58.963767 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.963677    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj" event={"ID":"e8bb2593-5acc-40c9-9025-e5e6b29bd80b","Type":"ContainerDied","Data":"f6313501eac3e27eb555b6e5c311f8385406512bb62587cddfdbb177af894661"}
Oct 29 22:11:58.968230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.963768    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj" event={"ID":"e8bb2593-5acc-40c9-9025-e5e6b29bd80b","Type":"ContainerDied","Data":"79333484e6cbe877a4409469a5e6d99281375ae2beb11a3f17d6aacfdb3682f7"}
Oct 29 22:11:58.968230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.964452    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"
Oct 29 22:11:58.968230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.966029    4550 generic.go:356] "Generic (PLEG): container finished" podID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerID="909c1cfa9b74bf779dad64d01d06fc589c01d337c386a8526095d7d53fd69fca" exitCode=0
Oct 29 22:11:58.968230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.966094    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" event={"ID":"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb","Type":"ContainerDied","Data":"909c1cfa9b74bf779dad64d01d06fc589c01d337c386a8526095d7d53fd69fca"}
Oct 29 22:11:58.968230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.966088    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"
Oct 29 22:11:58.968230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.966126    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" event={"ID":"f1fefa99-0a5c-422e-b950-89e0a7f2c1cb","Type":"ContainerDied","Data":"c723e071611310a1e078209d9d28e997a10e9969a71c655591514894e19aa5c5"}
Oct 29 22:11:58.968230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.967680    4550 scope.go:122] "RemoveContainer" containerID="34d3593ee7810d93590353aefc91ba1dad61c15c9a3a7b49ec8514e771475898"
Oct 29 22:11:58.968230 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:11:58.967993    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d3593ee7810d93590353aefc91ba1dad61c15c9a3a7b49ec8514e771475898\": container with ID starting with 34d3593ee7810d93590353aefc91ba1dad61c15c9a3a7b49ec8514e771475898 not found: ID does not exist" containerID="34d3593ee7810d93590353aefc91ba1dad61c15c9a3a7b49ec8514e771475898"
Oct 29 22:11:58.968230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.968019    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d3593ee7810d93590353aefc91ba1dad61c15c9a3a7b49ec8514e771475898"} err="failed to get container status \"34d3593ee7810d93590353aefc91ba1dad61c15c9a3a7b49ec8514e771475898\": rpc error: code = NotFound desc = could not find container \"34d3593ee7810d93590353aefc91ba1dad61c15c9a3a7b49ec8514e771475898\": container with ID starting with 34d3593ee7810d93590353aefc91ba1dad61c15c9a3a7b49ec8514e771475898 not found: ID does not exist"
Oct 29 22:11:58.968230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.968036    4550 scope.go:122] "RemoveContainer" containerID="f6313501eac3e27eb555b6e5c311f8385406512bb62587cddfdbb177af894661"
Oct 29 22:11:58.978989 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.978963    4550 scope.go:122] "RemoveContainer" containerID="f6313501eac3e27eb555b6e5c311f8385406512bb62587cddfdbb177af894661"
Oct 29 22:11:58.992214 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:11:58.980724    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6313501eac3e27eb555b6e5c311f8385406512bb62587cddfdbb177af894661\": container with ID starting with f6313501eac3e27eb555b6e5c311f8385406512bb62587cddfdbb177af894661 not found: ID does not exist" containerID="f6313501eac3e27eb555b6e5c311f8385406512bb62587cddfdbb177af894661"
Oct 29 22:11:58.992214 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.980781    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6313501eac3e27eb555b6e5c311f8385406512bb62587cddfdbb177af894661"} err="failed to get container status \"f6313501eac3e27eb555b6e5c311f8385406512bb62587cddfdbb177af894661\": rpc error: code = NotFound desc = could not find container \"f6313501eac3e27eb555b6e5c311f8385406512bb62587cddfdbb177af894661\": container with ID starting with f6313501eac3e27eb555b6e5c311f8385406512bb62587cddfdbb177af894661 not found: ID does not exist"
Oct 29 22:11:58.992214 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:58.980810    4550 scope.go:122] "RemoveContainer" containerID="909c1cfa9b74bf779dad64d01d06fc589c01d337c386a8526095d7d53fd69fca"
Oct 29 22:11:59.001014 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:59.000988    4550 scope.go:122] "RemoveContainer" containerID="909c1cfa9b74bf779dad64d01d06fc589c01d337c386a8526095d7d53fd69fca"
Oct 29 22:11:59.001410 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:11:59.001385    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909c1cfa9b74bf779dad64d01d06fc589c01d337c386a8526095d7d53fd69fca\": container with ID starting with 909c1cfa9b74bf779dad64d01d06fc589c01d337c386a8526095d7d53fd69fca not found: ID does not exist" containerID="909c1cfa9b74bf779dad64d01d06fc589c01d337c386a8526095d7d53fd69fca"
Oct 29 22:11:59.001477 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:59.001419    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909c1cfa9b74bf779dad64d01d06fc589c01d337c386a8526095d7d53fd69fca"} err="failed to get container status \"909c1cfa9b74bf779dad64d01d06fc589c01d337c386a8526095d7d53fd69fca\": rpc error: code = NotFound desc = could not find container \"909c1cfa9b74bf779dad64d01d06fc589c01d337c386a8526095d7d53fd69fca\": container with ID starting with 909c1cfa9b74bf779dad64d01d06fc589c01d337c386a8526095d7d53fd69fca not found: ID does not exist"
Oct 29 22:11:59.023194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:59.023163    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"]
Oct 29 22:11:59.029205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:59.029166    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-dn2lp"]
Oct 29 22:11:59.076950 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:59.076920    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"]
Oct 29 22:11:59.082889 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:59.082863    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2"]
Oct 29 22:11:59.118080 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:59.118053    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"]
Oct 29 22:11:59.121672 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:59.121647    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-kjqbj"]
Oct 29 22:11:59.527890 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:59.527757    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:11:59.563541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:59.563479    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:59.564072 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:59.563918    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-czpj2" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:59.991112 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:59.991079    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe0d723-a137-4c93-9657-ca9258b08a27" path="/var/lib/kubelet/pods/7fe0d723-a137-4c93-9657-ca9258b08a27/volumes"
Oct 29 22:11:59.991713 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:59.991694    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8bb2593-5acc-40c9-9025-e5e6b29bd80b" path="/var/lib/kubelet/pods/e8bb2593-5acc-40c9-9025-e5e6b29bd80b/volumes"
Oct 29 22:11:59.992237 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:11:59.992220    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" path="/var/lib/kubelet/pods/f1fefa99-0a5c-422e-b950-89e0a7f2c1cb/volumes"
Oct 29 22:12:00.323562 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.323485    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"]
Oct 29 22:12:00.324128 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.324114    4550 cpu_manager.go:414] "containerMap: removing container" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerName="istio-proxy"
Oct 29 22:12:00.324246 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.324130    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerName="istio-proxy"
Oct 29 22:12:00.324246 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.324159    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7fe0d723-a137-4c93-9657-ca9258b08a27" containerName="istio-proxy"
Oct 29 22:12:00.324246 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.324164    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe0d723-a137-4c93-9657-ca9258b08a27" containerName="istio-proxy"
Oct 29 22:12:00.324246 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.324178    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e8bb2593-5acc-40c9-9025-e5e6b29bd80b" containerName="istio-proxy"
Oct 29 22:12:00.324246 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.324186    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bb2593-5acc-40c9-9025-e5e6b29bd80b" containerName="istio-proxy"
Oct 29 22:12:00.324404 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.324297    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1fefa99-0a5c-422e-b950-89e0a7f2c1cb" containerName="istio-proxy"
Oct 29 22:12:00.324404 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.324308    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe0d723-a137-4c93-9657-ca9258b08a27" containerName="istio-proxy"
Oct 29 22:12:00.324404 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.324316    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bb2593-5acc-40c9-9025-e5e6b29bd80b" containerName="istio-proxy"
Oct 29 22:12:00.329885 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.329862    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.338229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.338205    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-podinfo\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.338347 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.338268    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-credential-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.338347 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.338307    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-envoy\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.338347 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.338345    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkczg\" (UniqueName: \"kubernetes.io/projected/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-kube-api-access-xkczg\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.338524 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.338372    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-data\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.338524 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.338425    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istiod-ca-cert\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.338524 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.338456    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-workload-certs\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.338524 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.338506    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-token\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.338749 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.338544    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-workload-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.345350 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.345329    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f82846-openshift-default-dockercfg-9s8bb\""
Oct 29 22:12:00.361584 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.361543    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"]
Oct 29 22:12:00.439959 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.439918    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-podinfo\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.440193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.439974    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-credential-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.440193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.440017    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-envoy\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.440193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.440064    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkczg\" (UniqueName: \"kubernetes.io/projected/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-kube-api-access-xkczg\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.440193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.440097    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-data\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.440193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.440186    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istiod-ca-cert\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.440567 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.440223    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-workload-certs\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.440567 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.440289    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-token\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.440567 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.440343    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-workload-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.440762 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.440670    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-workload-certs\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.440826 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.440764    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-workload-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.440885 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.440856    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-data\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.440980 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.440960    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istiod-ca-cert\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.441109 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.441081    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-credential-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.442910 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.442886    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-envoy\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.443131 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.443100    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-podinfo\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.463614 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.463584    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-token\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.464029 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.464008    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkczg\" (UniqueName: \"kubernetes.io/projected/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-kube-api-access-xkczg\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.648196 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.648164    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:00.982564 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:12:00.982532    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc8e407_edc2_4c71_ba8b_f43f53dc8f57.slice/crio-2f5b08172731bb9b5e3049d66628cf6bd9b9ce9cce30a6d93d1a4557bb454278 WatchSource:0}: Error finding container 2f5b08172731bb9b5e3049d66628cf6bd9b9ce9cce30a6d93d1a4557bb454278: Status 404 returned error can't find the container with id 2f5b08172731bb9b5e3049d66628cf6bd9b9ce9cce30a6d93d1a4557bb454278
Oct 29 22:12:00.982708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.982690    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"]
Oct 29 22:12:00.985519 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.985476    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:12:00.985647 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.985560    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:12:00.985647 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:00.985609    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:12:01.619515 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:01.619478    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:02.001524 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:02.001444    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv" event={"ID":"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57","Type":"ContainerStarted","Data":"002db6aa7a44b24b5284cd2641c51d422d74e904cb0c1d61f41dd82ca2d6f5b5"}
Oct 29 22:12:02.001524 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:02.001493    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv" event={"ID":"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57","Type":"ContainerStarted","Data":"2f5b08172731bb9b5e3049d66628cf6bd9b9ce9cce30a6d93d1a4557bb454278"}
Oct 29 22:12:02.059971 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:02.059919    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv" podStartSLOduration=2.059906585 podStartE2EDuration="2.059906585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:02.055526959 +0000 UTC m=+1538.212652458" watchObservedRunningTime="2026-10-29 22:12:02.059906585 +0000 UTC m=+1538.217032082"
Oct 29 22:12:02.649298 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:02.649268    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:02.651368 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:02.651344    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:12:02.651479 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:02.651391    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv" podUID="7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 29 22:12:03.424245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:03.424212    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:03.648434 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:03.648403    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:12:03.648604 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:03.648459    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv" podUID="7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 29 22:12:03.897671 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:03.897642    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"
Oct 29 22:12:04.649426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:04.649392    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:12:04.649603 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:04.649455    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv" podUID="7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 29 22:12:05.228311 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:05.228273    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:05.648975 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:05.648942    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:12:05.649131 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:05.649009    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv" podUID="7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 29 22:12:06.648678 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:06.648643    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:12:06.649105 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:06.648708    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv" podUID="7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 29 22:12:07.295006 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:07.294969    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:07.653290 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:07.653255    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:07.653690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:07.653606    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:07.654177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:07.654156    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:12:08.609196 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:08.609126    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:09.484024 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.483991    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"]
Oct 29 22:12:09.487985 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.487963    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.531818 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.531783    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"]
Oct 29 22:12:09.592965 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.592936    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-48c52b-openshift-default-dockercfg-zsvgm\""
Oct 29 22:12:09.639979 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.639948    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.640202 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.640094    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmsp\" (UniqueName: \"kubernetes.io/projected/e983d6ab-b49f-4e25-ba45-37f08b410917-kube-api-access-6zmsp\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.640271 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.640198    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e983d6ab-b49f-4e25-ba45-37f08b410917-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.640271 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.640220    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.640271 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.640250    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.640438 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.640424    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.640562 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.640543    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.640627 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.640608    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.640690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.640652    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.742648 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.742569    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.742808 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.742654    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.742808 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.742744    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.742808 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.742778    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.742808 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.742800    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.743032 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.742895    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmsp\" (UniqueName: \"kubernetes.io/projected/e983d6ab-b49f-4e25-ba45-37f08b410917-kube-api-access-6zmsp\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.743032 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.742941    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e983d6ab-b49f-4e25-ba45-37f08b410917-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.743032 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.742968    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.743225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.743056    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.743225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.743175    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.743225 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.743212    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.743353 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.743277    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.743527 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.743509    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.743859 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.743836    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e983d6ab-b49f-4e25-ba45-37f08b410917-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.745119 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.745098    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.745379 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.745362    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.765831 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.765805    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.765932 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.765884    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmsp\" (UniqueName: \"kubernetes.io/projected/e983d6ab-b49f-4e25-ba45-37f08b410917-kube-api-access-6zmsp\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:09.804471 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:09.804445    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:10.126240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:10.126206    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"]
Oct 29 22:12:10.127059 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:12:10.127032    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode983d6ab_b49f_4e25_ba45_37f08b410917.slice/crio-4fd652f48b6f05e3fb5882f0aeed8fbd2e1b3c719039aca0396140cd2ccda4c5 WatchSource:0}: Error finding container 4fd652f48b6f05e3fb5882f0aeed8fbd2e1b3c719039aca0396140cd2ccda4c5: Status 404 returned error can't find the container with id 4fd652f48b6f05e3fb5882f0aeed8fbd2e1b3c719039aca0396140cd2ccda4c5
Oct 29 22:12:10.130171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:10.130109    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:12:10.130375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:10.130205    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:12:10.130375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:10.130247    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:12:10.709358 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:10.709322    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:11.094052 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:11.094019    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v" event={"ID":"e983d6ab-b49f-4e25-ba45-37f08b410917","Type":"ContainerStarted","Data":"dcb9ee2a52be56b6bd68d748ee4d7831861db6242b706ed549a9e2583ba60ffb"}
Oct 29 22:12:11.094251 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:11.094074    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v" event={"ID":"e983d6ab-b49f-4e25-ba45-37f08b410917","Type":"ContainerStarted","Data":"4fd652f48b6f05e3fb5882f0aeed8fbd2e1b3c719039aca0396140cd2ccda4c5"}
Oct 29 22:12:11.152046 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:11.151988    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v" podStartSLOduration=2.151968925 podStartE2EDuration="2.151968925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:11.150972915 +0000 UTC m=+1547.308098432" watchObservedRunningTime="2026-10-29 22:12:11.151968925 +0000 UTC m=+1547.309094424"
Oct 29 22:12:11.513571 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:12:11.513471    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:12:11.804987 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:11.804906    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:11.806464 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:11.806437    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 29 22:12:11.806588 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:11.806510    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v" podUID="e983d6ab-b49f-4e25-ba45-37f08b410917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 29 22:12:12.399047 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:12.399013    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:12.805646 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:12.805572    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 29 22:12:12.806019 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:12.805641    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v" podUID="e983d6ab-b49f-4e25-ba45-37f08b410917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 29 22:12:13.805550 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:13.805517    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 29 22:12:13.805732 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:13.805569    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v" podUID="e983d6ab-b49f-4e25-ba45-37f08b410917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 29 22:12:13.977165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:13.973756    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:14.094725 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:14.094692    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:14.805715 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:14.805681    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 29 22:12:14.805892 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:14.805735    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v" podUID="e983d6ab-b49f-4e25-ba45-37f08b410917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 29 22:12:15.786412 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:15.786366    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:15.804983 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:15.804952    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 29 22:12:15.805185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:15.805012    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v" podUID="e983d6ab-b49f-4e25-ba45-37f08b410917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 29 22:12:15.910968 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:15.910932    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:16.810252 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:16.810224    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:16.810671 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:16.810652    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:16.811587 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:16.811563    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:12:17.653155 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:17.653108    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:20.007660 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:20.007627    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:21.923733 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:21.923690    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:23.661488 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:23.661456    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:25.931989 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:25.931946    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:27.189324 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:27.189292    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"]
Oct 29 22:12:27.189746 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:27.189500    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs" podUID="3b9c227f-4871-457c-a0fd-5893b6bcd0ba" containerName="maas-api" containerID="cri-o://12d58eb5b42e2ad632726b8ac42c80e70bc57ba41e023a79ee05b03768777628" gracePeriod=30
Oct 29 22:12:27.226433 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:27.226391    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:27.364713 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:27.364616    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:27.483965 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:27.483826    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:27.571329 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:27.571302    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:12:27.747925 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:27.747827    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-kube-api-access-pvjlc\" (UniqueName: \"kubernetes.io/projected/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-kube-api-access-pvjlc\") pod \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") "
Oct 29 22:12:27.759329 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:27.759298    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-maas-api-tls\") pod \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") "
Oct 29 22:12:27.760237 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:27.760207    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-metrics-certs\") pod \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") "
Oct 29 22:12:27.760674 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:27.760651    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-openshift-service-ca\") pod \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\" (UID: \"3b9c227f-4871-457c-a0fd-5893b6bcd0ba\") "
Oct 29 22:12:27.781043 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:27.781009    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-openshift-service-ca" pod "3b9c227f-4871-457c-a0fd-5893b6bcd0ba" (UID: "3b9c227f-4871-457c-a0fd-5893b6bcd0ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:27.799413 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:27.799369    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:27.889863 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:27.889102    4550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-openshift-service-ca\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:27.944162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:27.932876    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:28.033289 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.033231    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-kube-api-access-pvjlc" pod "3b9c227f-4871-457c-a0fd-5893b6bcd0ba" (UID: "3b9c227f-4871-457c-a0fd-5893b6bcd0ba"). InnerVolumeSpecName "kube-api-access-pvjlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:28.034894 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.034857    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-maas-api-tls" pod "3b9c227f-4871-457c-a0fd-5893b6bcd0ba" (UID: "3b9c227f-4871-457c-a0fd-5893b6bcd0ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:12:28.086813 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.086771    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-metrics-certs" pod "3b9c227f-4871-457c-a0fd-5893b6bcd0ba" (UID: "3b9c227f-4871-457c-a0fd-5893b6bcd0ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:12:28.095737 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.095704    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvjlc\" (UniqueName: \"kubernetes.io/projected/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-kube-api-access-pvjlc\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:28.096127 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.096103    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-maas-api-tls\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:28.096289 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.096273    4550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b9c227f-4871-457c-a0fd-5893b6bcd0ba-metrics-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:28.261775 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.261674    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:28.327855 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.327809    4550 generic.go:356] "Generic (PLEG): container finished" podID="3b9c227f-4871-457c-a0fd-5893b6bcd0ba" containerID="12d58eb5b42e2ad632726b8ac42c80e70bc57ba41e023a79ee05b03768777628" exitCode=0
Oct 29 22:12:28.328333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.328294    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs" event={"ID":"3b9c227f-4871-457c-a0fd-5893b6bcd0ba","Type":"ContainerDied","Data":"12d58eb5b42e2ad632726b8ac42c80e70bc57ba41e023a79ee05b03768777628"}
Oct 29 22:12:28.328569 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.328532    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs" event={"ID":"3b9c227f-4871-457c-a0fd-5893b6bcd0ba","Type":"ContainerDied","Data":"72025eaaaf41d0ffcccd57356fecadd5e0bc969168a7b81d59f36627104116f6"}
Oct 29 22:12:28.328737 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.328724    4550 scope.go:122] "RemoveContainer" containerID="12d58eb5b42e2ad632726b8ac42c80e70bc57ba41e023a79ee05b03768777628"
Oct 29 22:12:28.329189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.329169    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"
Oct 29 22:12:28.372611 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.372583    4550 scope.go:122] "RemoveContainer" containerID="12d58eb5b42e2ad632726b8ac42c80e70bc57ba41e023a79ee05b03768777628"
Oct 29 22:12:28.374593 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:12:28.374489    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d58eb5b42e2ad632726b8ac42c80e70bc57ba41e023a79ee05b03768777628\": container with ID starting with 12d58eb5b42e2ad632726b8ac42c80e70bc57ba41e023a79ee05b03768777628 not found: ID does not exist" containerID="12d58eb5b42e2ad632726b8ac42c80e70bc57ba41e023a79ee05b03768777628"
Oct 29 22:12:28.374593 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.374529    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d58eb5b42e2ad632726b8ac42c80e70bc57ba41e023a79ee05b03768777628"} err="failed to get container status \"12d58eb5b42e2ad632726b8ac42c80e70bc57ba41e023a79ee05b03768777628\": rpc error: code = NotFound desc = could not find container \"12d58eb5b42e2ad632726b8ac42c80e70bc57ba41e023a79ee05b03768777628\": container with ID starting with 12d58eb5b42e2ad632726b8ac42c80e70bc57ba41e023a79ee05b03768777628 not found: ID does not exist"
Oct 29 22:12:28.402639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.402606    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"]
Oct 29 22:12:28.414063 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.414028    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-edc2d3-7988f966c7-2jffs"]
Oct 29 22:12:28.931316 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:28.931274    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:29.048492 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:29.048460    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:29.502320 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:29.502292    4550 scope.go:122] "RemoveContainer" containerID="f78516fa9673215b6a174da2a012ac130f30c890196675b8f9928f6806cb664d"
Oct 29 22:12:29.580952 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:29.580920    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:29.991714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:29.991684    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b9c227f-4871-457c-a0fd-5893b6bcd0ba" path="/var/lib/kubelet/pods/3b9c227f-4871-457c-a0fd-5893b6bcd0ba/volumes"
Oct 29 22:12:30.821290 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:30.821259    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:32.438943 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:32.438911    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:34.122436 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:34.122401    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:35.596112 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:35.596083    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:35.718207 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:35.718173    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:35.847451 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:35.847365    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:35.918180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:35.916509    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"]
Oct 29 22:12:35.918180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:35.916796    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85" podUID="a9089a59-c618-4d5c-8957-9d095dd04959" containerName="istio-proxy" containerID="cri-o://fb71b5a29fc96f1e8ce9853c914d790ba0c389ae742053b8968cf25e7775f1ec" gracePeriod=30
Oct 29 22:12:35.930257 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:35.921129    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"]
Oct 29 22:12:35.930257 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:35.921344    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" containerID="cri-o://c3da622d4ea8ce35ebd711300c51c6568c59484aab09d40673508f6cb28c51cd" gracePeriod=30
Oct 29 22:12:35.930257 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:35.926041    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"]
Oct 29 22:12:35.930257 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:35.926244    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg" podUID="fd31e322-3948-4e25-998f-be538665aa9e" containerName="istio-proxy" containerID="cri-o://7fb3fa4d762d99e52b943142e4ca83a678c9bc5633419ae3edc8287aa0d6be12" gracePeriod=30
Oct 29 22:12:35.972453 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:35.972367    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:36.988360 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:36.988286    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:37.186193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:37.186127    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:37.312656 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:37.312564    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"]
Oct 29 22:12:37.312817 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:37.312774    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" containerID="cri-o://b3c6b3db187f2c39510c7a0d8ed423d7026c018c09abae1bc3c267cadf1372b2" gracePeriod=30
Oct 29 22:12:37.384780 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:37.384669    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:38.046826 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:38.046783    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:38.346244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:38.346196    4550 patch_prober.go:29] interesting pod/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:12:38.346521 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:38.346296    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:12:39.586214 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.586179    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"]
Oct 29 22:12:39.586846 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.586831    4550 cpu_manager.go:414] "containerMap: removing container" podUID="3b9c227f-4871-457c-a0fd-5893b6bcd0ba" containerName="maas-api"
Oct 29 22:12:39.586904 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.586847    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9c227f-4871-457c-a0fd-5893b6bcd0ba" containerName="maas-api"
Oct 29 22:12:39.586980 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.586971    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b9c227f-4871-457c-a0fd-5893b6bcd0ba" containerName="maas-api"
Oct 29 22:12:39.591918 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.591901    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.628006 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.627960    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"]
Oct 29 22:12:39.648593 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.648551    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.648780 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.648665    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.648780 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.648702    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.648780 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.648750    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.648958 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.648879    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.649535 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.649408    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.649535 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.649479    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.652718 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.649667    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj5pn\" (UniqueName: \"kubernetes.io/projected/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-kube-api-access-zj5pn\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.652718 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.649761    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.747553 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.747510    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:39.751002 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.750976    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.751119 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.751040    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.751119 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.751063    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.751119 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.751093    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.751295 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.751127    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.751295 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.751240    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.751295 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.751279    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.751450 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.751380    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj5pn\" (UniqueName: \"kubernetes.io/projected/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-kube-api-access-zj5pn\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.751517 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.751482    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.751610 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.751591    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.751679 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.751635    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.751743 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.751710    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.751843 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.751822    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.752321 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.752299    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.753631 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.753606    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.754554 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.754529    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.791573 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.784442    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.822994 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.822956    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj5pn\" (UniqueName: \"kubernetes.io/projected/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-kube-api-access-zj5pn\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:39.908074 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:39.907990    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:40.236482 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:40.236451    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"]
Oct 29 22:12:40.237532 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:12:40.237500    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e00a1d_038e_4ed0_8a8c_a85ae64e5452.slice/crio-e1cdd84e67725359b1aa19f8590ace372d94229de4293659bcbd0eac0e186e04 WatchSource:0}: Error finding container e1cdd84e67725359b1aa19f8590ace372d94229de4293659bcbd0eac0e186e04: Status 404 returned error can't find the container with id e1cdd84e67725359b1aa19f8590ace372d94229de4293659bcbd0eac0e186e04
Oct 29 22:12:40.240289 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:40.240254    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:12:40.240401 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:40.240332    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:12:40.240401 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:40.240374    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:12:40.419792 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:40.419755    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79" event={"ID":"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452","Type":"ContainerStarted","Data":"508a7a1d9c3d6061d4939fd2d3ade7f41e619a6a4cf6c7293a8c81cb601c6288"}
Oct 29 22:12:40.420006 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:40.419820    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79" event={"ID":"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452","Type":"ContainerStarted","Data":"e1cdd84e67725359b1aa19f8590ace372d94229de4293659bcbd0eac0e186e04"}
Oct 29 22:12:40.468374 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:40.468315    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79" podStartSLOduration=1.468297494 podStartE2EDuration="1.468297494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:40.465785019 +0000 UTC m=+1576.622910541" watchObservedRunningTime="2026-10-29 22:12:40.468297494 +0000 UTC m=+1576.625422996"
Oct 29 22:12:40.566218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:40.566120    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:12:40.566218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:40.566193    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:12:40.908099 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:40.908064    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:41.038059 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.038025    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:41.171291 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.170386    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:41.332205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.318792    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:12:41.371580 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.371552    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:12:41.405390 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.405357    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-envoy\") pod \"25bb493a-96b9-4ddf-af48-ee7eee126bef\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") "
Oct 29 22:12:41.405589 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.405403    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-workload-socket\") pod \"25bb493a-96b9-4ddf-af48-ee7eee126bef\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") "
Oct 29 22:12:41.405589 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.405469    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-podinfo\") pod \"25bb493a-96b9-4ddf-af48-ee7eee126bef\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") "
Oct 29 22:12:41.405589 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.405520    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-credential-socket\") pod \"25bb493a-96b9-4ddf-af48-ee7eee126bef\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") "
Oct 29 22:12:41.405765 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.405631    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25bb493a-96b9-4ddf-af48-ee7eee126bef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25bb493a-96b9-4ddf-af48-ee7eee126bef-istiod-ca-cert\") pod \"25bb493a-96b9-4ddf-af48-ee7eee126bef\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") "
Oct 29 22:12:41.405765 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.405680    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-data\") pod \"25bb493a-96b9-4ddf-af48-ee7eee126bef\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") "
Oct 29 22:12:41.405765 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.405723    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-workload-certs\") pod \"25bb493a-96b9-4ddf-af48-ee7eee126bef\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") "
Oct 29 22:12:41.405918 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.405768    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25bb493a-96b9-4ddf-af48-ee7eee126bef-kube-api-access-mh7zv\" (UniqueName: \"kubernetes.io/projected/25bb493a-96b9-4ddf-af48-ee7eee126bef-kube-api-access-mh7zv\") pod \"25bb493a-96b9-4ddf-af48-ee7eee126bef\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") "
Oct 29 22:12:41.405918 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.405801    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-token\" (UniqueName: \"kubernetes.io/projected/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-token\") pod \"25bb493a-96b9-4ddf-af48-ee7eee126bef\" (UID: \"25bb493a-96b9-4ddf-af48-ee7eee126bef\") "
Oct 29 22:12:41.406772 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.406446    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-credential-socket" pod "25bb493a-96b9-4ddf-af48-ee7eee126bef" (UID: "25bb493a-96b9-4ddf-af48-ee7eee126bef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:41.408069 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.408042    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25bb493a-96b9-4ddf-af48-ee7eee126bef-istiod-ca-cert" pod "25bb493a-96b9-4ddf-af48-ee7eee126bef" (UID: "25bb493a-96b9-4ddf-af48-ee7eee126bef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:41.408688 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.408290    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-workload-certs" pod "25bb493a-96b9-4ddf-af48-ee7eee126bef" (UID: "25bb493a-96b9-4ddf-af48-ee7eee126bef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:41.408688 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.408350    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-data" pod "25bb493a-96b9-4ddf-af48-ee7eee126bef" (UID: "25bb493a-96b9-4ddf-af48-ee7eee126bef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:41.408688 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.408661    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-workload-socket" pod "25bb493a-96b9-4ddf-af48-ee7eee126bef" (UID: "25bb493a-96b9-4ddf-af48-ee7eee126bef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:41.408903 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.408841    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.408903 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.408863    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.408903 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.408879    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25bb493a-96b9-4ddf-af48-ee7eee126bef-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.408903 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.408893    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.409047 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.408906    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.410162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.410108    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-podinfo" pod "25bb493a-96b9-4ddf-af48-ee7eee126bef" (UID: "25bb493a-96b9-4ddf-af48-ee7eee126bef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:41.411006 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.410976    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-token" pod "25bb493a-96b9-4ddf-af48-ee7eee126bef" (UID: "25bb493a-96b9-4ddf-af48-ee7eee126bef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:41.411164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.411119    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-envoy" pod "25bb493a-96b9-4ddf-af48-ee7eee126bef" (UID: "25bb493a-96b9-4ddf-af48-ee7eee126bef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:41.411631 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.411603    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25bb493a-96b9-4ddf-af48-ee7eee126bef-kube-api-access-mh7zv" pod "25bb493a-96b9-4ddf-af48-ee7eee126bef" (UID: "25bb493a-96b9-4ddf-af48-ee7eee126bef"). InnerVolumeSpecName "kube-api-access-mh7zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:41.430354 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.430280    4550 generic.go:356] "Generic (PLEG): container finished" podID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerID="c3da622d4ea8ce35ebd711300c51c6568c59484aab09d40673508f6cb28c51cd" exitCode=0
Oct 29 22:12:41.430354 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.430316    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"
Oct 29 22:12:41.430532 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.430358    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" event={"ID":"25bb493a-96b9-4ddf-af48-ee7eee126bef","Type":"ContainerDied","Data":"c3da622d4ea8ce35ebd711300c51c6568c59484aab09d40673508f6cb28c51cd"}
Oct 29 22:12:41.430532 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.430459    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr" event={"ID":"25bb493a-96b9-4ddf-af48-ee7eee126bef","Type":"ContainerDied","Data":"c23bb6e464a3a0091ef55952e5ce7a9dbd396192456c3bd7693effdcbf216046"}
Oct 29 22:12:41.430532 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.430493    4550 scope.go:122] "RemoveContainer" containerID="c3da622d4ea8ce35ebd711300c51c6568c59484aab09d40673508f6cb28c51cd"
Oct 29 22:12:41.433311 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.433292    4550 generic.go:356] "Generic (PLEG): container finished" podID="a9089a59-c618-4d5c-8957-9d095dd04959" containerID="fb71b5a29fc96f1e8ce9853c914d790ba0c389ae742053b8968cf25e7775f1ec" exitCode=0
Oct 29 22:12:41.433433 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.433420    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85" event={"ID":"a9089a59-c618-4d5c-8957-9d095dd04959","Type":"ContainerDied","Data":"fb71b5a29fc96f1e8ce9853c914d790ba0c389ae742053b8968cf25e7775f1ec"}
Oct 29 22:12:41.435201 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.435172    4550 generic.go:356] "Generic (PLEG): container finished" podID="fd31e322-3948-4e25-998f-be538665aa9e" containerID="7fb3fa4d762d99e52b943142e4ca83a678c9bc5633419ae3edc8287aa0d6be12" exitCode=0
Oct 29 22:12:41.435300 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.435219    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"
Oct 29 22:12:41.435384 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.435366    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg" event={"ID":"fd31e322-3948-4e25-998f-be538665aa9e","Type":"ContainerDied","Data":"7fb3fa4d762d99e52b943142e4ca83a678c9bc5633419ae3edc8287aa0d6be12"}
Oct 29 22:12:41.435466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.435405    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg" event={"ID":"fd31e322-3948-4e25-998f-be538665aa9e","Type":"ContainerDied","Data":"8a90233b06a2ff94e0b678b82e4811ea3b17e860787f29af70afaf7e641c6f9e"}
Oct 29 22:12:41.445897 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.445876    4550 scope.go:122] "RemoveContainer" containerID="c3da622d4ea8ce35ebd711300c51c6568c59484aab09d40673508f6cb28c51cd"
Oct 29 22:12:41.446219 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:12:41.446193    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3da622d4ea8ce35ebd711300c51c6568c59484aab09d40673508f6cb28c51cd\": container with ID starting with c3da622d4ea8ce35ebd711300c51c6568c59484aab09d40673508f6cb28c51cd not found: ID does not exist" containerID="c3da622d4ea8ce35ebd711300c51c6568c59484aab09d40673508f6cb28c51cd"
Oct 29 22:12:41.446308 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.446231    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3da622d4ea8ce35ebd711300c51c6568c59484aab09d40673508f6cb28c51cd"} err="failed to get container status \"c3da622d4ea8ce35ebd711300c51c6568c59484aab09d40673508f6cb28c51cd\": rpc error: code = NotFound desc = could not find container \"c3da622d4ea8ce35ebd711300c51c6568c59484aab09d40673508f6cb28c51cd\": container with ID starting with c3da622d4ea8ce35ebd711300c51c6568c59484aab09d40673508f6cb28c51cd not found: ID does not exist"
Oct 29 22:12:41.446308 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.446257    4550 scope.go:122] "RemoveContainer" containerID="7fb3fa4d762d99e52b943142e4ca83a678c9bc5633419ae3edc8287aa0d6be12"
Oct 29 22:12:41.457009 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.456992    4550 scope.go:122] "RemoveContainer" containerID="7fb3fa4d762d99e52b943142e4ca83a678c9bc5633419ae3edc8287aa0d6be12"
Oct 29 22:12:41.457329 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:12:41.457306    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb3fa4d762d99e52b943142e4ca83a678c9bc5633419ae3edc8287aa0d6be12\": container with ID starting with 7fb3fa4d762d99e52b943142e4ca83a678c9bc5633419ae3edc8287aa0d6be12 not found: ID does not exist" containerID="7fb3fa4d762d99e52b943142e4ca83a678c9bc5633419ae3edc8287aa0d6be12"
Oct 29 22:12:41.457416 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.457339    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb3fa4d762d99e52b943142e4ca83a678c9bc5633419ae3edc8287aa0d6be12"} err="failed to get container status \"7fb3fa4d762d99e52b943142e4ca83a678c9bc5633419ae3edc8287aa0d6be12\": rpc error: code = NotFound desc = could not find container \"7fb3fa4d762d99e52b943142e4ca83a678c9bc5633419ae3edc8287aa0d6be12\": container with ID starting with 7fb3fa4d762d99e52b943142e4ca83a678c9bc5633419ae3edc8287aa0d6be12 not found: ID does not exist"
Oct 29 22:12:41.539988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.537825    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd31e322-3948-4e25-998f-be538665aa9e-istio-token\" (UniqueName: \"kubernetes.io/projected/fd31e322-3948-4e25-998f-be538665aa9e-istio-token\") pod \"fd31e322-3948-4e25-998f-be538665aa9e\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") "
Oct 29 22:12:41.539988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.537976    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-credential-socket\") pod \"fd31e322-3948-4e25-998f-be538665aa9e\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") "
Oct 29 22:12:41.539988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.538081    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-istio-envoy\") pod \"fd31e322-3948-4e25-998f-be538665aa9e\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") "
Oct 29 22:12:41.539988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.538210    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-istio-data\") pod \"fd31e322-3948-4e25-998f-be538665aa9e\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") "
Oct 29 22:12:41.539988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.538253    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd31e322-3948-4e25-998f-be538665aa9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd31e322-3948-4e25-998f-be538665aa9e-istiod-ca-cert\") pod \"fd31e322-3948-4e25-998f-be538665aa9e\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") "
Oct 29 22:12:41.539988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.538298    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd31e322-3948-4e25-998f-be538665aa9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd31e322-3948-4e25-998f-be538665aa9e-istio-podinfo\") pod \"fd31e322-3948-4e25-998f-be538665aa9e\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") "
Oct 29 22:12:41.539988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.538355    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd31e322-3948-4e25-998f-be538665aa9e-kube-api-access-49mrm\" (UniqueName: \"kubernetes.io/projected/fd31e322-3948-4e25-998f-be538665aa9e-kube-api-access-49mrm\") pod \"fd31e322-3948-4e25-998f-be538665aa9e\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") "
Oct 29 22:12:41.539988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.538462    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-workload-socket\") pod \"fd31e322-3948-4e25-998f-be538665aa9e\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") "
Oct 29 22:12:41.539988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.538489    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-workload-certs\") pod \"fd31e322-3948-4e25-998f-be538665aa9e\" (UID: \"fd31e322-3948-4e25-998f-be538665aa9e\") "
Oct 29 22:12:41.539988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.539320    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.539988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.539345    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.539988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.539359    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh7zv\" (UniqueName: \"kubernetes.io/projected/25bb493a-96b9-4ddf-af48-ee7eee126bef-kube-api-access-mh7zv\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.539988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.539372    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25bb493a-96b9-4ddf-af48-ee7eee126bef-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.544740 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.542280    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-workload-certs" pod "fd31e322-3948-4e25-998f-be538665aa9e" (UID: "fd31e322-3948-4e25-998f-be538665aa9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:41.544740 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.542502    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-istio-data" pod "fd31e322-3948-4e25-998f-be538665aa9e" (UID: "fd31e322-3948-4e25-998f-be538665aa9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:41.544740 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.542757    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd31e322-3948-4e25-998f-be538665aa9e-istiod-ca-cert" pod "fd31e322-3948-4e25-998f-be538665aa9e" (UID: "fd31e322-3948-4e25-998f-be538665aa9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:41.544740 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.544701    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-credential-socket" pod "fd31e322-3948-4e25-998f-be538665aa9e" (UID: "fd31e322-3948-4e25-998f-be538665aa9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:41.545167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.545109    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-workload-socket" pod "fd31e322-3948-4e25-998f-be538665aa9e" (UID: "fd31e322-3948-4e25-998f-be538665aa9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:41.676566 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.676525    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.676854 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.676831    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.676983 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.676972    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd31e322-3948-4e25-998f-be538665aa9e-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.677070 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.677060    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.677188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.677173    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.745935 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.745809    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-istio-envoy" pod "fd31e322-3948-4e25-998f-be538665aa9e" (UID: "fd31e322-3948-4e25-998f-be538665aa9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:41.749393 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.749310    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"]
Oct 29 22:12:41.754418 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.754369    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd31e322-3948-4e25-998f-be538665aa9e-istio-podinfo" pod "fd31e322-3948-4e25-998f-be538665aa9e" (UID: "fd31e322-3948-4e25-998f-be538665aa9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:41.755533 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.755489    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd31e322-3948-4e25-998f-be538665aa9e-istio-token" pod "fd31e322-3948-4e25-998f-be538665aa9e" (UID: "fd31e322-3948-4e25-998f-be538665aa9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:41.757276 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.757009    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-pdjrr"]
Oct 29 22:12:41.759532 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.759514    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd31e322-3948-4e25-998f-be538665aa9e-kube-api-access-49mrm" pod "fd31e322-3948-4e25-998f-be538665aa9e" (UID: "fd31e322-3948-4e25-998f-be538665aa9e"). InnerVolumeSpecName "kube-api-access-49mrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:41.802162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.797987    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd31e322-3948-4e25-998f-be538665aa9e-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.802162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.798023    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd31e322-3948-4e25-998f-be538665aa9e-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.802162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.798039    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49mrm\" (UniqueName: \"kubernetes.io/projected/fd31e322-3948-4e25-998f-be538665aa9e-kube-api-access-49mrm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.802162 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.798055    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd31e322-3948-4e25-998f-be538665aa9e-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.917377 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.915411    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:41.919606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.917990    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 29 22:12:41.919606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:41.918067    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79" podUID="c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 29 22:12:41.921287 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:12:41.921198    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:12:42.062228 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.061133    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" path="/var/lib/kubelet/pods/25bb493a-96b9-4ddf-af48-ee7eee126bef/volumes"
Oct 29 22:12:42.134852 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.134777    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"]
Oct 29 22:12:42.138658 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.138624    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-jqftg"]
Oct 29 22:12:42.145048 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.145022    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:12:42.226913 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.226880    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9089a59-c618-4d5c-8957-9d095dd04959-kube-api-access-2846k\" (UniqueName: \"kubernetes.io/projected/a9089a59-c618-4d5c-8957-9d095dd04959-kube-api-access-2846k\") pod \"a9089a59-c618-4d5c-8957-9d095dd04959\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") "
Oct 29 22:12:42.227111 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.227015    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-workload-socket\") pod \"a9089a59-c618-4d5c-8957-9d095dd04959\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") "
Oct 29 22:12:42.227111 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.227088    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-istio-envoy\") pod \"a9089a59-c618-4d5c-8957-9d095dd04959\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") "
Oct 29 22:12:42.227260 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.227159    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9089a59-c618-4d5c-8957-9d095dd04959-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9089a59-c618-4d5c-8957-9d095dd04959-istio-podinfo\") pod \"a9089a59-c618-4d5c-8957-9d095dd04959\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") "
Oct 29 22:12:42.227260 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.227190    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9089a59-c618-4d5c-8957-9d095dd04959-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9089a59-c618-4d5c-8957-9d095dd04959-istiod-ca-cert\") pod \"a9089a59-c618-4d5c-8957-9d095dd04959\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") "
Oct 29 22:12:42.227260 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.227223    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-workload-certs\") pod \"a9089a59-c618-4d5c-8957-9d095dd04959\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") "
Oct 29 22:12:42.227411 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.227300    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-istio-data\") pod \"a9089a59-c618-4d5c-8957-9d095dd04959\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") "
Oct 29 22:12:42.227411 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.227347    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-credential-socket\") pod \"a9089a59-c618-4d5c-8957-9d095dd04959\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") "
Oct 29 22:12:42.227411 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.227380    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9089a59-c618-4d5c-8957-9d095dd04959-istio-token\" (UniqueName: \"kubernetes.io/projected/a9089a59-c618-4d5c-8957-9d095dd04959-istio-token\") pod \"a9089a59-c618-4d5c-8957-9d095dd04959\" (UID: \"a9089a59-c618-4d5c-8957-9d095dd04959\") "
Oct 29 22:12:42.227723 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.227694    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-workload-socket" pod "a9089a59-c618-4d5c-8957-9d095dd04959" (UID: "a9089a59-c618-4d5c-8957-9d095dd04959"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.228157 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.227889    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9089a59-c618-4d5c-8957-9d095dd04959-istiod-ca-cert" pod "a9089a59-c618-4d5c-8957-9d095dd04959" (UID: "a9089a59-c618-4d5c-8957-9d095dd04959"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:42.228157 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.228134    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-istio-data" pod "a9089a59-c618-4d5c-8957-9d095dd04959" (UID: "a9089a59-c618-4d5c-8957-9d095dd04959"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.228368 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.228346    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-credential-socket" pod "a9089a59-c618-4d5c-8957-9d095dd04959" (UID: "a9089a59-c618-4d5c-8957-9d095dd04959"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.228440 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.228367    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-workload-certs" pod "a9089a59-c618-4d5c-8957-9d095dd04959" (UID: "a9089a59-c618-4d5c-8957-9d095dd04959"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.228693 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.228674    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.228734 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.228698    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9089a59-c618-4d5c-8957-9d095dd04959-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.228734 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.228713    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.228734 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.228726    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.228827 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.228739    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.230040 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.229996    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9089a59-c618-4d5c-8957-9d095dd04959-kube-api-access-2846k" pod "a9089a59-c618-4d5c-8957-9d095dd04959" (UID: "a9089a59-c618-4d5c-8957-9d095dd04959"). InnerVolumeSpecName "kube-api-access-2846k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.230040 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.229997    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9089a59-c618-4d5c-8957-9d095dd04959-istio-podinfo" pod "a9089a59-c618-4d5c-8957-9d095dd04959" (UID: "a9089a59-c618-4d5c-8957-9d095dd04959"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:42.230247 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.230076    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-istio-envoy" pod "a9089a59-c618-4d5c-8957-9d095dd04959" (UID: "a9089a59-c618-4d5c-8957-9d095dd04959"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.231478 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.231449    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9089a59-c618-4d5c-8957-9d095dd04959-istio-token" pod "a9089a59-c618-4d5c-8957-9d095dd04959" (UID: "a9089a59-c618-4d5c-8957-9d095dd04959"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.329946 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.329909    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2846k\" (UniqueName: \"kubernetes.io/projected/a9089a59-c618-4d5c-8957-9d095dd04959-kube-api-access-2846k\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.330131 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.329950    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9089a59-c618-4d5c-8957-9d095dd04959-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.330131 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.329968    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9089a59-c618-4d5c-8957-9d095dd04959-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.330131 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.329980    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9089a59-c618-4d5c-8957-9d095dd04959-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.445690 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.445663    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"
Oct 29 22:12:42.445817 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.445668    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85" event={"ID":"a9089a59-c618-4d5c-8957-9d095dd04959","Type":"ContainerDied","Data":"996d631e108eb1ec262814260d127e9905e82b9eb717ab43a389f9b2432dffec"}
Oct 29 22:12:42.445868 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.445845    4550 scope.go:122] "RemoveContainer" containerID="fb71b5a29fc96f1e8ce9853c914d790ba0c389ae742053b8968cf25e7775f1ec"
Oct 29 22:12:42.448515 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.448487    4550 generic.go:356] "Generic (PLEG): container finished" podID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerID="b3c6b3db187f2c39510c7a0d8ed423d7026c018c09abae1bc3c267cadf1372b2" exitCode=0
Oct 29 22:12:42.448628 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.448575    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" event={"ID":"aec85ac8-9f07-402f-98d3-cb8b5f5fe581","Type":"ContainerDied","Data":"b3c6b3db187f2c39510c7a0d8ed423d7026c018c09abae1bc3c267cadf1372b2"}
Oct 29 22:12:42.457109 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.457091    4550 scope.go:122] "RemoveContainer" containerID="1b27f106dfc515e505cf13888d762b8646cb0fb0d18481889c1465b92b5398c0"
Oct 29 22:12:42.505751 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.505719    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"]
Oct 29 22:12:42.510000 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.509975    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4nr85"]
Oct 29 22:12:42.618832 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.618804    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:12:42.735365 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.735331    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istiod-ca-cert\") pod \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") "
Oct 29 22:12:42.735552 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.735516    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-podinfo\") pod \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") "
Oct 29 22:12:42.735630 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.735554    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-workload-socket\") pod \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") "
Oct 29 22:12:42.735630 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.735600    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-kube-api-access-mpns5\" (UniqueName: \"kubernetes.io/projected/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-kube-api-access-mpns5\") pod \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") "
Oct 29 22:12:42.735758 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.735649    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-data\") pod \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") "
Oct 29 22:12:42.735817 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.735791    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-token\" (UniqueName: \"kubernetes.io/projected/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-token\") pod \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") "
Oct 29 22:12:42.735870 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.735829    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-envoy\") pod \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") "
Oct 29 22:12:42.735930 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.735867    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-workload-socket" pod "aec85ac8-9f07-402f-98d3-cb8b5f5fe581" (UID: "aec85ac8-9f07-402f-98d3-cb8b5f5fe581"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.735930 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.735890    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-credential-socket\") pod \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") "
Oct 29 22:12:42.735930 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.735894    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istiod-ca-cert" pod "aec85ac8-9f07-402f-98d3-cb8b5f5fe581" (UID: "aec85ac8-9f07-402f-98d3-cb8b5f5fe581"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:42.736099 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.735929    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-workload-certs\") pod \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\" (UID: \"aec85ac8-9f07-402f-98d3-cb8b5f5fe581\") "
Oct 29 22:12:42.736099 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.736015    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-data" pod "aec85ac8-9f07-402f-98d3-cb8b5f5fe581" (UID: "aec85ac8-9f07-402f-98d3-cb8b5f5fe581"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.736317 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.736096    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-credential-socket" pod "aec85ac8-9f07-402f-98d3-cb8b5f5fe581" (UID: "aec85ac8-9f07-402f-98d3-cb8b5f5fe581"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.736380 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.736268    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-workload-certs" pod "aec85ac8-9f07-402f-98d3-cb8b5f5fe581" (UID: "aec85ac8-9f07-402f-98d3-cb8b5f5fe581"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.736738 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.736713    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.736845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.736743    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.736845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.736759    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.736845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.736774    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.736845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.736791    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.743286 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.743222    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-podinfo" pod "aec85ac8-9f07-402f-98d3-cb8b5f5fe581" (UID: "aec85ac8-9f07-402f-98d3-cb8b5f5fe581"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:42.743403 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.743294    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-envoy" pod "aec85ac8-9f07-402f-98d3-cb8b5f5fe581" (UID: "aec85ac8-9f07-402f-98d3-cb8b5f5fe581"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.743403 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.743336    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-kube-api-access-mpns5" pod "aec85ac8-9f07-402f-98d3-cb8b5f5fe581" (UID: "aec85ac8-9f07-402f-98d3-cb8b5f5fe581"). InnerVolumeSpecName "kube-api-access-mpns5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.743403 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.743360    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-token" pod "aec85ac8-9f07-402f-98d3-cb8b5f5fe581" (UID: "aec85ac8-9f07-402f-98d3-cb8b5f5fe581"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.838674 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.838645    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.838825 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.838675    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpns5\" (UniqueName: \"kubernetes.io/projected/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-kube-api-access-mpns5\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.838825 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.838697    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.838825 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.838706    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec85ac8-9f07-402f-98d3-cb8b5f5fe581-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.908603 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.908526    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 29 22:12:42.908603 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:42.908587    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79" podUID="c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 29 22:12:43.459456 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:43.459429    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"
Oct 29 22:12:43.459897 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:43.459427    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc" event={"ID":"aec85ac8-9f07-402f-98d3-cb8b5f5fe581","Type":"ContainerDied","Data":"3ad44b053879c8b12c892ec9602a4de46d5b646c039e5e242fb528270ea3c81e"}
Oct 29 22:12:43.459897 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:43.459576    4550 scope.go:122] "RemoveContainer" containerID="b3c6b3db187f2c39510c7a0d8ed423d7026c018c09abae1bc3c267cadf1372b2"
Oct 29 22:12:43.534750 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:43.534719    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"]
Oct 29 22:12:43.545307 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:43.545273    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9ae129-openshift-default-65fc746555-g28qc"]
Oct 29 22:12:43.819574 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:43.819507    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:43.922201 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:43.919823    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 29 22:12:43.922201 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:43.919885    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79" podUID="c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 29 22:12:44.006264 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:44.006235    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9089a59-c618-4d5c-8957-9d095dd04959" path="/var/lib/kubelet/pods/a9089a59-c618-4d5c-8957-9d095dd04959/volumes"
Oct 29 22:12:44.006782 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:44.006763    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" path="/var/lib/kubelet/pods/aec85ac8-9f07-402f-98d3-cb8b5f5fe581/volumes"
Oct 29 22:12:44.007251 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:44.007237    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd31e322-3948-4e25-998f-be538665aa9e" path="/var/lib/kubelet/pods/fd31e322-3948-4e25-998f-be538665aa9e/volumes"
Oct 29 22:12:44.909045 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:44.909010    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 29 22:12:44.909441 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:44.909066    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79" podUID="c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 29 22:12:45.606649 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.606615    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"]
Oct 29 22:12:45.607342 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.607326    4550 cpu_manager.go:414] "containerMap: removing container" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy"
Oct 29 22:12:45.607409 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.607343    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy"
Oct 29 22:12:45.607409 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.607358    4550 cpu_manager.go:414] "containerMap: removing container" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy"
Oct 29 22:12:45.607409 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.607363    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy"
Oct 29 22:12:45.607409 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.607395    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a9089a59-c618-4d5c-8957-9d095dd04959" containerName="istio-proxy"
Oct 29 22:12:45.607409 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.607401    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9089a59-c618-4d5c-8957-9d095dd04959" containerName="istio-proxy"
Oct 29 22:12:45.607568 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.607427    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fd31e322-3948-4e25-998f-be538665aa9e" containerName="istio-proxy"
Oct 29 22:12:45.607568 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.607432    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd31e322-3948-4e25-998f-be538665aa9e" containerName="istio-proxy"
Oct 29 22:12:45.607568 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.607528    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9089a59-c618-4d5c-8957-9d095dd04959" containerName="istio-proxy"
Oct 29 22:12:45.607568 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.607542    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd31e322-3948-4e25-998f-be538665aa9e" containerName="istio-proxy"
Oct 29 22:12:45.607568 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.607549    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy"
Oct 29 22:12:45.607568 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.607555    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy"
Oct 29 22:12:45.607568 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.607562    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="25bb493a-96b9-4ddf-af48-ee7eee126bef" containerName="istio-proxy"
Oct 29 22:12:45.611668 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.611089    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.617754 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.617728    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"]
Oct 29 22:12:45.727035 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.727001    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-workload-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.727246 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.727055    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fae4096-9b8a-4938-82b4-098312132fe7-istio-token\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.727297 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.727269    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-istio-data\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.727333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.727297    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fae4096-9b8a-4938-82b4-098312132fe7-istio-podinfo\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.727333 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.727329    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-credential-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.727433 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.727382    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-workload-certs\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.727517 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.727501    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fae4096-9b8a-4938-82b4-098312132fe7-istiod-ca-cert\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.727560 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.727548    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nztrw\" (UniqueName: \"kubernetes.io/projected/7fae4096-9b8a-4938-82b4-098312132fe7-kube-api-access-nztrw\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.727601 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.727572    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-istio-envoy\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.828883 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.828843    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-istio-data\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.829071 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.828890    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fae4096-9b8a-4938-82b4-098312132fe7-istio-podinfo\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.829071 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.829006    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-credential-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.829071 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.829041    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-workload-certs\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.829268 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.829129    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:45.829268 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.829185    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fae4096-9b8a-4938-82b4-098312132fe7-istiod-ca-cert\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.829268 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.829251    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nztrw\" (UniqueName: \"kubernetes.io/projected/7fae4096-9b8a-4938-82b4-098312132fe7-kube-api-access-nztrw\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.829512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.829265    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-istio-data\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.829512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.829283    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-istio-envoy\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.829512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.829420    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-workload-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.829512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.829442    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-credential-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.829512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.829460    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-workload-certs\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.829512 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.829499    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fae4096-9b8a-4938-82b4-098312132fe7-istio-token\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.829812 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.829707    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-workload-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.830032 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.830008    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fae4096-9b8a-4938-82b4-098312132fe7-istiod-ca-cert\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.831575 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.831550    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-istio-envoy\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.831772 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.831753    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fae4096-9b8a-4938-82b4-098312132fe7-istio-podinfo\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.852267 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.852226    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fae4096-9b8a-4938-82b4-098312132fe7-istio-token\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.853041 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.853015    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nztrw\" (UniqueName: \"kubernetes.io/projected/7fae4096-9b8a-4938-82b4-098312132fe7-kube-api-access-nztrw\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:45.923164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.916824    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 29 22:12:45.923164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.916883    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79" podUID="c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 29 22:12:45.933224 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:45.932181    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:46.312493 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:46.312468    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"]
Oct 29 22:12:46.312986 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:12:46.312962    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fae4096_9b8a_4938_82b4_098312132fe7.slice/crio-a2422fe0ce646cf4605bdc1c06fac29d883dc4f2146d22a0ca405c98bb759909 WatchSource:0}: Error finding container a2422fe0ce646cf4605bdc1c06fac29d883dc4f2146d22a0ca405c98bb759909: Status 404 returned error can't find the container with id a2422fe0ce646cf4605bdc1c06fac29d883dc4f2146d22a0ca405c98bb759909
Oct 29 22:12:46.315705 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:46.315676    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:12:46.315817 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:46.315759    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:12:46.315817 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:46.315809    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:12:46.598335 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:46.598303    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8" event={"ID":"7fae4096-9b8a-4938-82b4-098312132fe7","Type":"ContainerStarted","Data":"0ab68a045916e9b8ba839d83b427e4395814e8883abc4b03e00e7e38ee109dfd"}
Oct 29 22:12:46.598548 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:46.598370    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8" event={"ID":"7fae4096-9b8a-4938-82b4-098312132fe7","Type":"ContainerStarted","Data":"a2422fe0ce646cf4605bdc1c06fac29d883dc4f2146d22a0ca405c98bb759909"}
Oct 29 22:12:46.647280 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:46.647220    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8" podStartSLOduration=1.647202472 podStartE2EDuration="1.647202472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:46.643375301 +0000 UTC m=+1582.800500819" watchObservedRunningTime="2026-10-29 22:12:46.647202472 +0000 UTC m=+1582.804327971"
Oct 29 22:12:46.908495 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:46.908398    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 29 22:12:46.908495 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:46.908462    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79" podUID="c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 29 22:12:46.933094 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:46.933064    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:47.913842 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:47.913810    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:47.914190 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:47.914171    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:47.914955 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:47.914933    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:12:47.935325 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:47.935290    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:12:47.935715 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:47.935342    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8" podUID="7fae4096-9b8a-4938-82b4-098312132fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 29 22:12:48.410081 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:48.410008    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:48.521642 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:48.521597    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:48.732004 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:48.731908    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:48.933563 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:48.933526    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:12:48.933751 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:48.933586    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8" podUID="7fae4096-9b8a-4938-82b4-098312132fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 29 22:12:49.933499 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:49.933465    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:12:49.933930 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:49.933514    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8" podUID="7fae4096-9b8a-4938-82b4-098312132fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 29 22:12:50.582978 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:50.582940    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"]
Oct 29 22:12:50.713216 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:50.713183    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:50.831955 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:50.831921    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:50.936073 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:50.935939    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:12:50.937054 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:50.937020    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8" podUID="7fae4096-9b8a-4938-82b4-098312132fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 29 22:12:50.973716 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:50.973678    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:51.933385 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:51.933348    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:12:51.933590 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:51.933404    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8" podUID="7fae4096-9b8a-4938-82b4-098312132fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 29 22:12:52.880167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:52.879662    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:52.933074 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:52.933049    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:12:52.933292 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:52.933271    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8" podUID="7fae4096-9b8a-4938-82b4-098312132fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 29 22:12:54.794117 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:54.794081    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:54.797210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:54.797191    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:55.052817 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.052727    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:12:55.112167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.097310    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"]
Oct 29 22:12:55.112167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.098429    4550 cpu_manager.go:414] "containerMap: removing container" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy"
Oct 29 22:12:55.112167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.098454    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec85ac8-9f07-402f-98d3-cb8b5f5fe581" containerName="istio-proxy"
Oct 29 22:12:55.195320 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.158444    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.373195 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.338738    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.373195 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.338808    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.373195 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.338844    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.373195 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.338873    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.373195 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.338901    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.373195 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.339003    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5hrj\" (UniqueName: \"kubernetes.io/projected/08a13f88-6791-4bd9-a15f-55b80bbee84d-kube-api-access-j5hrj\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.373195 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.339078    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a13f88-6791-4bd9-a15f-55b80bbee84d-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.373195 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.339117    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.376017 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.373930    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.448578 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.421591    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"]
Oct 29 22:12:55.504166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.482084    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5hrj\" (UniqueName: \"kubernetes.io/projected/08a13f88-6791-4bd9-a15f-55b80bbee84d-kube-api-access-j5hrj\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.506485 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.506454    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a13f88-6791-4bd9-a15f-55b80bbee84d-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.506761 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.506744    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.518941 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.518900    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.519390 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.519361    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.520708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.520121    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.520708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.519777    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.520708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.520179    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.520708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.519962    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.520708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.507164    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.520708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.520212    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.520708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.520244    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.520708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.507271    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a13f88-6791-4bd9-a15f-55b80bbee84d-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.521242 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.520741    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.558985 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.558949    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5hrj\" (UniqueName: \"kubernetes.io/projected/08a13f88-6791-4bd9-a15f-55b80bbee84d-kube-api-access-j5hrj\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.607168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.588375    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:55.607168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.589220    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5" podUID="8020c6d3-1330-4581-8824-2ca1bdf0bc35" containerName="payload-processing" containerID="cri-o://057e534b1b2915fa8ccd7cf5c28a62bfc635a63797b948c2ab62d352286c8996" gracePeriod=25
Oct 29 22:12:55.608860 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.608825    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.623192 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.611889    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.623809 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.623708    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:55.702435 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.690255    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:55.702435 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.698256    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:55.737767 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:55.737469    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:12:56.036166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:56.022647    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:12:56.352165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:56.346245    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:56.424959 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:56.424887    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:12:56.591531 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:56.589974    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:56.591531 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:56.590494    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:12:56.736173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:56.735320    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:56.736173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:56.735658    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:12:56.736808 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:56.736745    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:12:57.012214 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:57.012129    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:12:57.012547 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:57.012519    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:12:57.012806 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:57.012778    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:12:57.063172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:57.037585    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:57.065836 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:57.065801    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:12:57.473458 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:57.444815    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:57.473458 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:57.464751    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:12:57.915061 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:57.915004    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:12:57.916630 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:57.916582    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:12:57.948245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:57.948193    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:57.963953 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:57.963909    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:58.087265 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:58.038359    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:12:58.241447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:58.189006    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:12:58.241447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:58.189329    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:12:58.241447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:58.189783    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:58.241447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:58.217933    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Oct 29 22:12:58.690193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:58.686560    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc" event={"ID":"08a13f88-6791-4bd9-a15f-55b80bbee84d","Type":"ContainerStarted","Data":"7d19890f36c0d0c3b4901094a96a0517464269b086f40dc8bb3705755234fc0c"}
Oct 29 22:12:58.690193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:58.686717    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:12:58.690193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:58.686781    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:12:58.690193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:58.686810    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"]
Oct 29 22:12:58.740844 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:58.734073    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:12:58.767215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:58.744565    4550 generic.go:356] "Generic (PLEG): container finished" podID="8020c6d3-1330-4581-8824-2ca1bdf0bc35" containerID="057e534b1b2915fa8ccd7cf5c28a62bfc635a63797b948c2ab62d352286c8996" exitCode=0
Oct 29 22:12:58.767215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:58.752252    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5" event={"ID":"8020c6d3-1330-4581-8824-2ca1bdf0bc35","Type":"ContainerDied","Data":"057e534b1b2915fa8ccd7cf5c28a62bfc635a63797b948c2ab62d352286c8996"}
Oct 29 22:12:58.770744 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:58.770618    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"
Oct 29 22:12:58.901357 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:58.879223    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8020c6d3-1330-4581-8824-2ca1bdf0bc35-kube-api-access-tkhgv\" (UniqueName: \"kubernetes.io/projected/8020c6d3-1330-4581-8824-2ca1bdf0bc35-kube-api-access-tkhgv\") pod \"8020c6d3-1330-4581-8824-2ca1bdf0bc35\" (UID: \"8020c6d3-1330-4581-8824-2ca1bdf0bc35\") "
Oct 29 22:12:58.901357 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:58.879317    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8020c6d3-1330-4581-8824-2ca1bdf0bc35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8020c6d3-1330-4581-8824-2ca1bdf0bc35-plugins-config-volume\") pod \"8020c6d3-1330-4581-8824-2ca1bdf0bc35\" (UID: \"8020c6d3-1330-4581-8824-2ca1bdf0bc35\") "
Oct 29 22:12:59.213168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.192474    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:59.223211 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.220267    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:12:59.232339 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.228881    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8020c6d3-1330-4581-8824-2ca1bdf0bc35-plugins-config-volume" pod "8020c6d3-1330-4581-8824-2ca1bdf0bc35" (UID: "8020c6d3-1330-4581-8824-2ca1bdf0bc35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:59.316379 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.316288    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:12:59.317060 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.317027    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:12:59.361486 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.361429    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:59.362304 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.362259    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:12:59.390359 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.363381    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:59.390359 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.366455    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:59.390359 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.390283    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8020c6d3-1330-4581-8824-2ca1bdf0bc35-kube-api-access-tkhgv" pod "8020c6d3-1330-4581-8824-2ca1bdf0bc35" (UID: "8020c6d3-1330-4581-8824-2ca1bdf0bc35"). InnerVolumeSpecName "kube-api-access-tkhgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:59.391467 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.391319    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:12:59.391467 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.391368    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:12:59.391467 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.391419    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:12:59.556288 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.556254    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8020c6d3-1330-4581-8824-2ca1bdf0bc35-plugins-config-volume\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:59.556511 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.556495    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkhgv\" (UniqueName: \"kubernetes.io/projected/8020c6d3-1330-4581-8824-2ca1bdf0bc35-kube-api-access-tkhgv\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:12:59.585679 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.585640    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.012223 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.971958    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"
Oct 29 22:13:00.068168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:12:59.973246    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5" event={"ID":"8020c6d3-1330-4581-8824-2ca1bdf0bc35","Type":"ContainerDied","Data":"ae0047ebebbf60abc4d31a62dfd87846afaefdfcf56c2feaddc741ec437987ec"}
Oct 29 22:13:00.068168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:00.022526    4550 scope.go:122] "RemoveContainer" containerID="057e534b1b2915fa8ccd7cf5c28a62bfc635a63797b948c2ab62d352286c8996"
Oct 29 22:13:00.227121 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:00.209809    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.241648 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:00.230268    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.321632 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:00.321348    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.321809 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:00.321649    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:13:00.379208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:00.369250    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.379208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:00.369596    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.379208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:00.370084    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:13:00.379208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:00.370427    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.379208 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:00.370620    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:13:00.525293 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:00.525244    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 29 22:13:00.862168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:00.859201    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"]
Oct 29 22:13:00.918613 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:00.918577    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75db5b-696cb4656d-5mkm5"]
Oct 29 22:13:01.072135 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.072092    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" podUID="93b4187a-1551-4cf8-b807-d057baf03920" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.095505 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.095465    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc" event={"ID":"08a13f88-6791-4bd9-a15f-55b80bbee84d","Type":"ContainerStarted","Data":"a84b32722bf3b4ae4acb0ff4947c03a91f272f0ece6920561b1743f62625c755"}
Oct 29 22:13:01.200226 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.199708    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc" podStartSLOduration=7.19968508 podStartE2EDuration="7.19968508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:01.193708033 +0000 UTC m=+1597.350833532" watchObservedRunningTime="2026-10-29 22:13:01.19968508 +0000 UTC m=+1597.356810582"
Oct 29 22:13:01.218178 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.216626    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.244303 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.244258    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.317205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.293267    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:13:01.328265 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.328216    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:13:01.351314 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.342676    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:13:01.442169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.411483    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:13:01.442408 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.413243    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.442611 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.413280    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:13:01.507233 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.499487    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" podUID="93b4187a-1551-4cf8-b807-d057baf03920" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.747169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.741383    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:13:01.747169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.743739    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 29 22:13:01.747169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.743820    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc" podUID="08a13f88-6791-4bd9-a15f-55b80bbee84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 29 22:13:01.993167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.992324    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:01.993167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:01.992388    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.039424 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:02.039388    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8020c6d3-1330-4581-8824-2ca1bdf0bc35" path="/var/lib/kubelet/pods/8020c6d3-1330-4581-8824-2ca1bdf0bc35/volumes"
Oct 29 22:13:02.230196 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:02.229345    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:02.230196 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:02.229405    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.230196 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:02.229805    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:13:02.247820 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:02.246913    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.293651 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:02.293610    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.351052 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:02.351007    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.351757 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:02.351316    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:13:02.457182 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:02.456243    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.461171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:02.457373    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.461171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:02.457708    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:13:02.733030 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:02.732978    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.754658 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:02.754617    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 29 22:13:02.754925 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:02.754894    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc" podUID="08a13f88-6791-4bd9-a15f-55b80bbee84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 29 22:13:03.244168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:03.243787    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.244749 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:03.244694    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:13:03.258597 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:03.258555    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.312207 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:03.304074    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 29 22:13:03.312898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:03.312871    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:13:03.352465 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:03.352420    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.352829 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:03.352802    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:13:03.475164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:03.460339    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.476382 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:03.475928    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.502226 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:03.475966    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.739860 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:03.739259    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 29 22:13:03.739860 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:03.739317    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc" podUID="08a13f88-6791-4bd9-a15f-55b80bbee84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 29 22:13:03.765482 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:03.753314    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:03.765482 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:03.753390    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.255193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:04.254602    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:13:04.266361 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:04.265386    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.399180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:04.380281    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:13:04.399180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:04.380930    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:13:04.399180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:04.383076    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.451616 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:04.451580    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:04.478066 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:04.478014    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.479595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:04.479546    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.520257 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:04.507876    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.744437 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:04.744403    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 29 22:13:04.744733 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:04.744704    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc" podUID="08a13f88-6791-4bd9-a15f-55b80bbee84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 29 22:13:04.757417 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:04.757388    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:13:05.076164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:05.064397    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:13:05.103755 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:05.085449    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:13:05.134847 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:05.132353    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:13:05.149348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:05.146775    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:13:05.254168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:05.251443    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:13:05.254943 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:05.254908    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:05.454400 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:05.454317    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:13:05.523560 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:05.513843    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:13:05.739537 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:05.739455    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 29 22:13:05.739537 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:05.739509    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc" podUID="08a13f88-6791-4bd9-a15f-55b80bbee84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 29 22:13:06.520016 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:06.519974    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:06.764293 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:06.748647    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 29 22:13:06.771671 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:06.771579    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc" podUID="08a13f88-6791-4bd9-a15f-55b80bbee84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 29 22:13:07.738378 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:07.738331    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 29 22:13:07.738832 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:07.738397    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc" podUID="08a13f88-6791-4bd9-a15f-55b80bbee84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 29 22:13:08.340842 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:08.340806    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:08.739114 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:08.739025    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 29 22:13:08.739114 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:08.739094    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc" podUID="08a13f88-6791-4bd9-a15f-55b80bbee84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 29 22:13:09.739374 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:09.739334    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 29 22:13:09.740033 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:09.739408    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc" podUID="08a13f88-6791-4bd9-a15f-55b80bbee84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 29 22:13:10.177302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:10.177240    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:10.739046 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:10.739012    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:13:10.744132 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:10.744101    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:13:11.493324 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:13:11.493290    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:13:11.568674 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:11.568633    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:13:12.017756 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.017729    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"]
Oct 29 22:13:12.020097 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.017963    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp" podUID="06aa5d20-0931-4e09-98ef-6b7cf4c1a59d" containerName="maas-api" containerID="cri-o://d8a85cdd2158e5ceb73a911d742b8e79b7597295b46b0caff5baf72880c63861" gracePeriod=30
Oct 29 22:13:12.093653 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.093602    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:12.643479 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.643449    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:13:12.722798 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.722767    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-metrics-certs\") pod \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") "
Oct 29 22:13:12.724024 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.724000    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-maas-api-tls\") pod \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") "
Oct 29 22:13:12.724408 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.724389    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-openshift-service-ca\") pod \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") "
Oct 29 22:13:12.724875 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.724821    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-openshift-service-ca" pod "06aa5d20-0931-4e09-98ef-6b7cf4c1a59d" (UID: "06aa5d20-0931-4e09-98ef-6b7cf4c1a59d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:12.725568 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.725451    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-kube-api-access-lk8xq\" (UniqueName: \"kubernetes.io/projected/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-kube-api-access-lk8xq\") pod \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\" (UID: \"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d\") "
Oct 29 22:13:12.726505 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.726483    4550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-openshift-service-ca\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:12.727754 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.727705    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-metrics-certs" pod "06aa5d20-0931-4e09-98ef-6b7cf4c1a59d" (UID: "06aa5d20-0931-4e09-98ef-6b7cf4c1a59d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:12.728001 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.727966    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-maas-api-tls" pod "06aa5d20-0931-4e09-98ef-6b7cf4c1a59d" (UID: "06aa5d20-0931-4e09-98ef-6b7cf4c1a59d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:12.730179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.730128    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-kube-api-access-lk8xq" pod "06aa5d20-0931-4e09-98ef-6b7cf4c1a59d" (UID: "06aa5d20-0931-4e09-98ef-6b7cf4c1a59d"). InnerVolumeSpecName "kube-api-access-lk8xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:12.731902 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.731049    4550 generic.go:356] "Generic (PLEG): container finished" podID="b27b23e3-07bd-4531-9d64-73d1b66de485" containerID="3b1a1e2579e8bf8fe938ff45559dae6bb7a27cd5f164817a07c4da911fa170cf" exitCode=0
Oct 29 22:13:12.732249 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.732073    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p" event={"ID":"b27b23e3-07bd-4531-9d64-73d1b66de485","Type":"ContainerDied","Data":"3b1a1e2579e8bf8fe938ff45559dae6bb7a27cd5f164817a07c4da911fa170cf"}
Oct 29 22:13:12.732862 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.732830    4550 scope.go:122] "RemoveContainer" containerID="3b1a1e2579e8bf8fe938ff45559dae6bb7a27cd5f164817a07c4da911fa170cf"
Oct 29 22:13:12.750381 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.744310    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:13:12.750381 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.744470    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:13:12.750381 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.744526    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 29 22:13:12.754184 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.753972    4550 generic.go:356] "Generic (PLEG): container finished" podID="06aa5d20-0931-4e09-98ef-6b7cf4c1a59d" containerID="d8a85cdd2158e5ceb73a911d742b8e79b7597295b46b0caff5baf72880c63861" exitCode=0
Oct 29 22:13:12.754184 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.754084    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp" event={"ID":"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d","Type":"ContainerDied","Data":"d8a85cdd2158e5ceb73a911d742b8e79b7597295b46b0caff5baf72880c63861"}
Oct 29 22:13:12.754391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.754185    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp" event={"ID":"06aa5d20-0931-4e09-98ef-6b7cf4c1a59d","Type":"ContainerDied","Data":"f5dde086ec241dc72e382b9ebccdd9b66706ed9f6af1272b494d4641a137abe1"}
Oct 29 22:13:12.754391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.754228    4550 scope.go:122] "RemoveContainer" containerID="d8a85cdd2158e5ceb73a911d742b8e79b7597295b46b0caff5baf72880c63861"
Oct 29 22:13:12.756118 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.756099    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"
Oct 29 22:13:12.834286 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.834231    4550 scope.go:122] "RemoveContainer" containerID="d8a85cdd2158e5ceb73a911d742b8e79b7597295b46b0caff5baf72880c63861"
Oct 29 22:13:12.834877 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.834847    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:12.836776 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:13:12.836718    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a85cdd2158e5ceb73a911d742b8e79b7597295b46b0caff5baf72880c63861\": container with ID starting with d8a85cdd2158e5ceb73a911d742b8e79b7597295b46b0caff5baf72880c63861 not found: ID does not exist" containerID="d8a85cdd2158e5ceb73a911d742b8e79b7597295b46b0caff5baf72880c63861"
Oct 29 22:13:12.836889 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.836780    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a85cdd2158e5ceb73a911d742b8e79b7597295b46b0caff5baf72880c63861"} err="failed to get container status \"d8a85cdd2158e5ceb73a911d742b8e79b7597295b46b0caff5baf72880c63861\": rpc error: code = NotFound desc = could not find container \"d8a85cdd2158e5ceb73a911d742b8e79b7597295b46b0caff5baf72880c63861\": container with ID starting with d8a85cdd2158e5ceb73a911d742b8e79b7597295b46b0caff5baf72880c63861 not found: ID does not exist"
Oct 29 22:13:12.839227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.838082    4550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-metrics-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:12.839227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.838168    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-maas-api-tls\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:12.839227 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.838223    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk8xq\" (UniqueName: \"kubernetes.io/projected/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d-kube-api-access-lk8xq\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:12.912201 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.911004    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"]
Oct 29 22:13:12.916519 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.916487    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp"]
Oct 29 22:13:12.974240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:12.974207    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:13.122765 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:13.122726    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:13.521062 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:13.521033    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:13.769795 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:13.769756    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p" event={"ID":"b27b23e3-07bd-4531-9d64-73d1b66de485","Type":"ContainerStarted","Data":"026efd88a94b6a2b6b5cec91b23e0891a54175126ba444ffede798b542f2635b"}
Oct 29 22:13:13.996262 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:13.996210    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06aa5d20-0931-4e09-98ef-6b7cf4c1a59d" path="/var/lib/kubelet/pods/06aa5d20-0931-4e09-98ef-6b7cf4c1a59d/volumes"
Oct 29 22:13:14.088693 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:14.088643    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:14.781165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:14.780586    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:13:14.781165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:14.780727    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:13:14.781165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:14.780771    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p" podUID="b27b23e3-07bd-4531-9d64-73d1b66de485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:13:15.035896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:15.035806    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:15.604257 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:15.604214    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-27cda7-6f9bcf964b-hwfqp" podUID="06aa5d20-0931-4e09-98ef-6b7cf4c1a59d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.109:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:15.784535 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:15.784496    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:13:15.785181 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:15.785121    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p" podUID="b27b23e3-07bd-4531-9d64-73d1b66de485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:13:16.769168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:16.768480    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:13:16.769168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:16.768542    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p" podUID="b27b23e3-07bd-4531-9d64-73d1b66de485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:13:16.886704 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:16.886668    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:17.442954 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:17.442868    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:17.769323 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:17.769235    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:13:17.769594 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:17.769563    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p" podUID="b27b23e3-07bd-4531-9d64-73d1b66de485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:13:17.802291 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:17.802261    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:13:18.223941 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:18.223898    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:18.407163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:18.406882    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:18.777165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:18.776382    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:13:18.777165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:18.776445    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p" podUID="b27b23e3-07bd-4531-9d64-73d1b66de485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:13:19.637595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:19.637559    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:19.805217 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:19.805122    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:13:19.807232 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:19.807204    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:13:20.158327 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:20.158297    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:20.280802 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:20.280757    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:20.557862 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:20.557775    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:21.848359 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:21.848302    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:22.014726 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:22.014678    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:22.147043 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:22.146958    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:22.411784 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:22.411644    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:22.713469 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:22.713313    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:22.868592 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:22.868555    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:23.281457 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:23.281428    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:24.631023 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:24.628354    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:26.803520 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:26.803486    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:28.414100 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:28.414066    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:29.960524 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:29.960432    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:30.076654 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:30.076605    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:30.199832 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:30.199785    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:30.419531 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:30.419500    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"]
Oct 29 22:13:30.419766 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:30.419738    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv" podUID="7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" containerName="istio-proxy" containerID="cri-o://002db6aa7a44b24b5284cd2641c51d422d74e904cb0c1d61f41dd82ca2d6f5b5" gracePeriod=30
Oct 29 22:13:30.425185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:30.425121    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"]
Oct 29 22:13:30.425402 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:30.425377    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8" podUID="7fae4096-9b8a-4938-82b4-098312132fe7" containerName="istio-proxy" containerID="cri-o://0ab68a045916e9b8ba839d83b427e4395814e8883abc4b03e00e7e38ee109dfd" gracePeriod=30
Oct 29 22:13:30.440489 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:30.440440    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:30.943255 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:30.943227    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:31.614491 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:31.614459    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:31.727412 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:31.727377    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:33.487175 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:33.487116    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:33.613391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:33.613361    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:33.821364 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:33.821289    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:35.685235 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.685091    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:35.747058 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.747025    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:13:35.818881 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.818849    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:13:35.873739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.869441    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-workload-socket\") pod \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") "
Oct 29 22:13:35.873739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.869499    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-token\" (UniqueName: \"kubernetes.io/projected/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-token\") pod \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") "
Oct 29 22:13:35.873739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.869536    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-kube-api-access-xkczg\" (UniqueName: \"kubernetes.io/projected/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-kube-api-access-xkczg\") pod \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") "
Oct 29 22:13:35.873739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.869630    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-workload-certs\") pod \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") "
Oct 29 22:13:35.873739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.869707    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-credential-socket\") pod \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") "
Oct 29 22:13:35.873739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.869831    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istiod-ca-cert\") pod \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") "
Oct 29 22:13:35.873739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.869873    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-podinfo\") pod \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") "
Oct 29 22:13:35.873739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.869940    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-data\") pod \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") "
Oct 29 22:13:35.873739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.870059    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-envoy\") pod \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\" (UID: \"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57\") "
Oct 29 22:13:35.963068 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.910297    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-workload-socket" pod "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" (UID: "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:35.963068 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.910944    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-envoy" pod "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" (UID: "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:35.963068 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.915224    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-data" pod "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" (UID: "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:35.963068 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.915449    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-workload-certs" pod "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" (UID: "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:35.963068 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.920058    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-token" pod "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" (UID: "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:35.963068 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.931364    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istiod-ca-cert" pod "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" (UID: "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.971480    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-workload-socket\") pod \"7fae4096-9b8a-4938-82b4-098312132fe7\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") "
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.971768    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-credential-socket\") pod \"7fae4096-9b8a-4938-82b4-098312132fe7\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") "
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.971825    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fae4096-9b8a-4938-82b4-098312132fe7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fae4096-9b8a-4938-82b4-098312132fe7-istio-podinfo\") pod \"7fae4096-9b8a-4938-82b4-098312132fe7\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") "
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.971861    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-workload-certs\") pod \"7fae4096-9b8a-4938-82b4-098312132fe7\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") "
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.971896    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fae4096-9b8a-4938-82b4-098312132fe7-kube-api-access-nztrw\" (UniqueName: \"kubernetes.io/projected/7fae4096-9b8a-4938-82b4-098312132fe7-kube-api-access-nztrw\") pod \"7fae4096-9b8a-4938-82b4-098312132fe7\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") "
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.971941    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fae4096-9b8a-4938-82b4-098312132fe7-istio-token\" (UniqueName: \"kubernetes.io/projected/7fae4096-9b8a-4938-82b4-098312132fe7-istio-token\") pod \"7fae4096-9b8a-4938-82b4-098312132fe7\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") "
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.971986    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-istio-envoy\") pod \"7fae4096-9b8a-4938-82b4-098312132fe7\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") "
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.972066    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fae4096-9b8a-4938-82b4-098312132fe7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fae4096-9b8a-4938-82b4-098312132fe7-istiod-ca-cert\") pod \"7fae4096-9b8a-4938-82b4-098312132fe7\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") "
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.972101    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-istio-data\") pod \"7fae4096-9b8a-4938-82b4-098312132fe7\" (UID: \"7fae4096-9b8a-4938-82b4-098312132fe7\") "
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.972445    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-workload-certs" pod "7fae4096-9b8a-4938-82b4-098312132fe7" (UID: "7fae4096-9b8a-4938-82b4-098312132fe7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.972662    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-workload-socket" pod "7fae4096-9b8a-4938-82b4-098312132fe7" (UID: "7fae4096-9b8a-4938-82b4-098312132fe7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.972820    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-credential-socket" pod "7fae4096-9b8a-4938-82b4-098312132fe7" (UID: "7fae4096-9b8a-4938-82b4-098312132fe7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.974510    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.974534    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.974549    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.974562    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.974575    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.974588    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.974602    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.974615    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.974631    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.974903    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-istio-data" pod "7fae4096-9b8a-4938-82b4-098312132fe7" (UID: "7fae4096-9b8a-4938-82b4-098312132fe7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:35.975472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:35.975223    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fae4096-9b8a-4938-82b4-098312132fe7-istiod-ca-cert" pod "7fae4096-9b8a-4938-82b4-098312132fe7" (UID: "7fae4096-9b8a-4938-82b4-098312132fe7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:36.133253 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.097387    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fae4096-9b8a-4938-82b4-098312132fe7-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.133253 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.097424    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.133253 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.098837    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:36.133253 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.115830    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-credential-socket" pod "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" (UID: "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.211307 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.193363    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fae4096-9b8a-4938-82b4-098312132fe7-istio-token" pod "7fae4096-9b8a-4938-82b4-098312132fe7" (UID: "7fae4096-9b8a-4938-82b4-098312132fe7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:36.211307 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.199375    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.211307 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.207269    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fae4096-9b8a-4938-82b4-098312132fe7-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.211307 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.206404    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-kube-api-access-xkczg" pod "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" (UID: "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57"). InnerVolumeSpecName "kube-api-access-xkczg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:36.211307 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.206470    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fae4096-9b8a-4938-82b4-098312132fe7-istio-podinfo" pod "7fae4096-9b8a-4938-82b4-098312132fe7" (UID: "7fae4096-9b8a-4938-82b4-098312132fe7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:13:36.211733 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.211712    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fae4096-9b8a-4938-82b4-098312132fe7-kube-api-access-nztrw" pod "7fae4096-9b8a-4938-82b4-098312132fe7" (UID: "7fae4096-9b8a-4938-82b4-098312132fe7"). InnerVolumeSpecName "kube-api-access-nztrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:36.254193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.225478    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-istio-envoy" pod "7fae4096-9b8a-4938-82b4-098312132fe7" (UID: "7fae4096-9b8a-4938-82b4-098312132fe7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.254193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.243274    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-podinfo" pod "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" (UID: "7cc8e407-edc2-4c71-ba8b-f43f53dc8f57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:13:36.327164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.324737    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"]
Oct 29 22:13:36.373129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.341116    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fae4096-9b8a-4938-82b4-098312132fe7-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.373129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.341165    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.373129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.341179    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkczg\" (UniqueName: \"kubernetes.io/projected/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57-kube-api-access-xkczg\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.373129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.341194    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fae4096-9b8a-4938-82b4-098312132fe7-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.373129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.341210    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nztrw\" (UniqueName: \"kubernetes.io/projected/7fae4096-9b8a-4938-82b4-098312132fe7-kube-api-access-nztrw\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.421708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.419718    4550 generic.go:356] "Generic (PLEG): container finished" podID="7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" containerID="002db6aa7a44b24b5284cd2641c51d422d74e904cb0c1d61f41dd82ca2d6f5b5" exitCode=0
Oct 29 22:13:36.421708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.419834    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv" event={"ID":"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57","Type":"ContainerDied","Data":"002db6aa7a44b24b5284cd2641c51d422d74e904cb0c1d61f41dd82ca2d6f5b5"}
Oct 29 22:13:36.421708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.419887    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv" event={"ID":"7cc8e407-edc2-4c71-ba8b-f43f53dc8f57","Type":"ContainerDied","Data":"2f5b08172731bb9b5e3049d66628cf6bd9b9ce9cce30a6d93d1a4557bb454278"}
Oct 29 22:13:36.421708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.419921    4550 scope.go:122] "RemoveContainer" containerID="002db6aa7a44b24b5284cd2641c51d422d74e904cb0c1d61f41dd82ca2d6f5b5"
Oct 29 22:13:36.421708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.420093    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"
Oct 29 22:13:36.441159 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.437095    4550 generic.go:356] "Generic (PLEG): container finished" podID="7fae4096-9b8a-4938-82b4-098312132fe7" containerID="0ab68a045916e9b8ba839d83b427e4395814e8883abc4b03e00e7e38ee109dfd" exitCode=0
Oct 29 22:13:36.441159 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.437210    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8" event={"ID":"7fae4096-9b8a-4938-82b4-098312132fe7","Type":"ContainerDied","Data":"0ab68a045916e9b8ba839d83b427e4395814e8883abc4b03e00e7e38ee109dfd"}
Oct 29 22:13:36.441159 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.437260    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8" event={"ID":"7fae4096-9b8a-4938-82b4-098312132fe7","Type":"ContainerDied","Data":"a2422fe0ce646cf4605bdc1c06fac29d883dc4f2146d22a0ca405c98bb759909"}
Oct 29 22:13:36.441159 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.437346    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"
Oct 29 22:13:36.485714 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.484403    4550 scope.go:122] "RemoveContainer" containerID="002db6aa7a44b24b5284cd2641c51d422d74e904cb0c1d61f41dd82ca2d6f5b5"
Oct 29 22:13:36.486056 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:13:36.486031    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002db6aa7a44b24b5284cd2641c51d422d74e904cb0c1d61f41dd82ca2d6f5b5\": container with ID starting with 002db6aa7a44b24b5284cd2641c51d422d74e904cb0c1d61f41dd82ca2d6f5b5 not found: ID does not exist" containerID="002db6aa7a44b24b5284cd2641c51d422d74e904cb0c1d61f41dd82ca2d6f5b5"
Oct 29 22:13:36.486314 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.486070    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002db6aa7a44b24b5284cd2641c51d422d74e904cb0c1d61f41dd82ca2d6f5b5"} err="failed to get container status \"002db6aa7a44b24b5284cd2641c51d422d74e904cb0c1d61f41dd82ca2d6f5b5\": rpc error: code = NotFound desc = could not find container \"002db6aa7a44b24b5284cd2641c51d422d74e904cb0c1d61f41dd82ca2d6f5b5\": container with ID starting with 002db6aa7a44b24b5284cd2641c51d422d74e904cb0c1d61f41dd82ca2d6f5b5 not found: ID does not exist"
Oct 29 22:13:36.486314 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.486097    4550 scope.go:122] "RemoveContainer" containerID="0ab68a045916e9b8ba839d83b427e4395814e8883abc4b03e00e7e38ee109dfd"
Oct 29 22:13:36.523197 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.523156    4550 scope.go:122] "RemoveContainer" containerID="0ab68a045916e9b8ba839d83b427e4395814e8883abc4b03e00e7e38ee109dfd"
Oct 29 22:13:36.523570 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:13:36.523541    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab68a045916e9b8ba839d83b427e4395814e8883abc4b03e00e7e38ee109dfd\": container with ID starting with 0ab68a045916e9b8ba839d83b427e4395814e8883abc4b03e00e7e38ee109dfd not found: ID does not exist" containerID="0ab68a045916e9b8ba839d83b427e4395814e8883abc4b03e00e7e38ee109dfd"
Oct 29 22:13:36.523682 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.523576    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab68a045916e9b8ba839d83b427e4395814e8883abc4b03e00e7e38ee109dfd"} err="failed to get container status \"0ab68a045916e9b8ba839d83b427e4395814e8883abc4b03e00e7e38ee109dfd\": rpc error: code = NotFound desc = could not find container \"0ab68a045916e9b8ba839d83b427e4395814e8883abc4b03e00e7e38ee109dfd\": container with ID starting with 0ab68a045916e9b8ba839d83b427e4395814e8883abc4b03e00e7e38ee109dfd not found: ID does not exist"
Oct 29 22:13:36.554524 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.554486    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"]
Oct 29 22:13:36.561280 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.561241    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-nlkdv"]
Oct 29 22:13:36.594743 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.594712    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"]
Oct 29 22:13:36.599476 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.599444    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-2lwl8"]
Oct 29 22:13:36.674595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.674559    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:36.792401 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.792325    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:36.939165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:36.936518    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:37.304518 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:37.304486    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:37.990183 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:37.990132    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" path="/var/lib/kubelet/pods/7cc8e407-edc2-4c71-ba8b-f43f53dc8f57/volumes"
Oct 29 22:13:37.990649 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:37.990635    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fae4096-9b8a-4938-82b4-098312132fe7" path="/var/lib/kubelet/pods/7fae4096-9b8a-4938-82b4-098312132fe7/volumes"
Oct 29 22:13:38.614734 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:38.614703    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:40.326391 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:40.326354    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:41.492457 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:13:41.492423    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:13:41.752230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:41.752131    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:41.877894 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:41.877799    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:42.485974 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:42.485937    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:44.146429 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:44.146399    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:44.261659 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:44.261625    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:44.382344 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:44.382307    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:44.475205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:44.472595    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"]
Oct 29 22:13:44.475205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:44.472850    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79" podUID="c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" containerName="istio-proxy" containerID="cri-o://508a7a1d9c3d6061d4939fd2d3ade7f41e619a6a4cf6c7293a8c81cb601c6288" gracePeriod=30
Oct 29 22:13:44.499427 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:44.499389    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"]
Oct 29 22:13:44.499994 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:44.499964    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v" podUID="e983d6ab-b49f-4e25-ba45-37f08b410917" containerName="istio-proxy" containerID="cri-o://dcb9ee2a52be56b6bd68d748ee4d7831861db6242b706ed549a9e2583ba60ffb" gracePeriod=30
Oct 29 22:13:44.505174 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:44.504528    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"]
Oct 29 22:13:44.505174 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:44.504743    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc" podUID="08a13f88-6791-4bd9-a15f-55b80bbee84d" containerName="istio-proxy" containerID="cri-o://a84b32722bf3b4ae4acb0ff4947c03a91f272f0ece6920561b1743f62625c755" gracePeriod=30
Oct 29 22:13:44.508278 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:44.508225    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:45.026907 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:45.026870    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:45.150293 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:45.150261    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:45.285326 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:45.285217    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:45.669896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:45.669851    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:46.813915 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:46.813880    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:13:46.814362 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:46.813938    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v" podUID="e983d6ab-b49f-4e25-ba45-37f08b410917" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:13:47.609088 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:47.609044    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:47.917121 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:47.917045    4550 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:13:47.917121 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:47.917099    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79" podUID="c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:13:49.424084 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:49.424029    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:49.838465 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:49.838437    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:13:49.891697 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:49.891675    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:13:49.895663 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:49.895638    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:13:49.964569 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:49.964483    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"]
Oct 29 22:13:49.977940 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:49.977908    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"]
Oct 29 22:13:50.009324 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009291    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-envoy\") pod \"e983d6ab-b49f-4e25-ba45-37f08b410917\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") "
Oct 29 22:13:50.009520 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009371    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-workload-socket\") pod \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") "
Oct 29 22:13:50.009520 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009428    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-credential-socket\") pod \"e983d6ab-b49f-4e25-ba45-37f08b410917\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") "
Oct 29 22:13:50.009520 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009454    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-token\" (UniqueName: \"kubernetes.io/projected/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-token\") pod \"e983d6ab-b49f-4e25-ba45-37f08b410917\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") "
Oct 29 22:13:50.009520 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009494    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-workload-socket\") pod \"e983d6ab-b49f-4e25-ba45-37f08b410917\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") "
Oct 29 22:13:50.009737 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009519    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-workload-certs\") pod \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") "
Oct 29 22:13:50.009737 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009546    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-podinfo\") pod \"08a13f88-6791-4bd9-a15f-55b80bbee84d\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") "
Oct 29 22:13:50.009737 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009606    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-workload-certs\") pod \"e983d6ab-b49f-4e25-ba45-37f08b410917\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") "
Oct 29 22:13:50.009737 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009636    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-data\") pod \"e983d6ab-b49f-4e25-ba45-37f08b410917\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") "
Oct 29 22:13:50.009737 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009674    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-workload-socket\") pod \"08a13f88-6791-4bd9-a15f-55b80bbee84d\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") "
Oct 29 22:13:50.009737 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009702    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e983d6ab-b49f-4e25-ba45-37f08b410917-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e983d6ab-b49f-4e25-ba45-37f08b410917-istiod-ca-cert\") pod \"e983d6ab-b49f-4e25-ba45-37f08b410917\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") "
Oct 29 22:13:50.010053 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009763    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-envoy\") pod \"08a13f88-6791-4bd9-a15f-55b80bbee84d\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") "
Oct 29 22:13:50.010053 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009842    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a13f88-6791-4bd9-a15f-55b80bbee84d-kube-api-access-j5hrj\" (UniqueName: \"kubernetes.io/projected/08a13f88-6791-4bd9-a15f-55b80bbee84d-kube-api-access-j5hrj\") pod \"08a13f88-6791-4bd9-a15f-55b80bbee84d\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") "
Oct 29 22:13:50.010053 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009887    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-credential-socket\") pod \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") "
Oct 29 22:13:50.010053 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009933    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-workload-certs\") pod \"08a13f88-6791-4bd9-a15f-55b80bbee84d\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") "
Oct 29 22:13:50.010053 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009961    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-credential-socket\") pod \"08a13f88-6791-4bd9-a15f-55b80bbee84d\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") "
Oct 29 22:13:50.010053 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.009991    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-podinfo\") pod \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") "
Oct 29 22:13:50.010053 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.010045    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e983d6ab-b49f-4e25-ba45-37f08b410917-kube-api-access-6zmsp\" (UniqueName: \"kubernetes.io/projected/e983d6ab-b49f-4e25-ba45-37f08b410917-kube-api-access-6zmsp\") pod \"e983d6ab-b49f-4e25-ba45-37f08b410917\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") "
Oct 29 22:13:50.010595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.010076    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-kube-api-access-zj5pn\" (UniqueName: \"kubernetes.io/projected/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-kube-api-access-zj5pn\") pod \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") "
Oct 29 22:13:50.010595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.010117    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-envoy\") pod \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") "
Oct 29 22:13:50.010595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.010175    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08a13f88-6791-4bd9-a15f-55b80bbee84d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a13f88-6791-4bd9-a15f-55b80bbee84d-istiod-ca-cert\") pod \"08a13f88-6791-4bd9-a15f-55b80bbee84d\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") "
Oct 29 22:13:50.010595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.010304    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-token\" (UniqueName: \"kubernetes.io/projected/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-token\") pod \"08a13f88-6791-4bd9-a15f-55b80bbee84d\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") "
Oct 29 22:13:50.010595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.010358    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-data\") pod \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") "
Oct 29 22:13:50.010595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.010389    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-data\") pod \"08a13f88-6791-4bd9-a15f-55b80bbee84d\" (UID: \"08a13f88-6791-4bd9-a15f-55b80bbee84d\") "
Oct 29 22:13:50.010595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.010467    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istiod-ca-cert\") pod \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") "
Oct 29 22:13:50.010595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.010511    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-token\" (UniqueName: \"kubernetes.io/projected/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-token\") pod \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\" (UID: \"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452\") "
Oct 29 22:13:50.010595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.010544    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-podinfo\") pod \"e983d6ab-b49f-4e25-ba45-37f08b410917\" (UID: \"e983d6ab-b49f-4e25-ba45-37f08b410917\") "
Oct 29 22:13:50.013353 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.013162    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-credential-socket" pod "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" (UID: "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.013703 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.013531    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-credential-socket" pod "08a13f88-6791-4bd9-a15f-55b80bbee84d" (UID: "08a13f88-6791-4bd9-a15f-55b80bbee84d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.014117 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.013844    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-workload-certs" pod "08a13f88-6791-4bd9-a15f-55b80bbee84d" (UID: "08a13f88-6791-4bd9-a15f-55b80bbee84d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.014569 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.014299    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-workload-certs" pod "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" (UID: "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.014939 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.014917    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a13f88-6791-4bd9-a15f-55b80bbee84d-istiod-ca-cert" pod "08a13f88-6791-4bd9-a15f-55b80bbee84d" (UID: "08a13f88-6791-4bd9-a15f-55b80bbee84d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:50.017736 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.015516    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-workload-socket" pod "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" (UID: "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.018237 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.015722    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-credential-socket" pod "e983d6ab-b49f-4e25-ba45-37f08b410917" (UID: "e983d6ab-b49f-4e25-ba45-37f08b410917"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.018371 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.016594    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-data" pod "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" (UID: "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.018477 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.016855    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-workload-certs" pod "e983d6ab-b49f-4e25-ba45-37f08b410917" (UID: "e983d6ab-b49f-4e25-ba45-37f08b410917"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.018586 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.017066    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-data" pod "e983d6ab-b49f-4e25-ba45-37f08b410917" (UID: "e983d6ab-b49f-4e25-ba45-37f08b410917"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.018701 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.017283    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-workload-socket" pod "08a13f88-6791-4bd9-a15f-55b80bbee84d" (UID: "08a13f88-6791-4bd9-a15f-55b80bbee84d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.018799 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.017586    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e983d6ab-b49f-4e25-ba45-37f08b410917-istiod-ca-cert" pod "e983d6ab-b49f-4e25-ba45-37f08b410917" (UID: "e983d6ab-b49f-4e25-ba45-37f08b410917"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:50.018894 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.018112    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-workload-socket" pod "e983d6ab-b49f-4e25-ba45-37f08b410917" (UID: "e983d6ab-b49f-4e25-ba45-37f08b410917"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.018984 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.018198    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istiod-ca-cert" pod "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" (UID: "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:50.019072 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.018272    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-podinfo" pod "e983d6ab-b49f-4e25-ba45-37f08b410917" (UID: "e983d6ab-b49f-4e25-ba45-37f08b410917"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:13:50.019182 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.018417    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-data" pod "08a13f88-6791-4bd9-a15f-55b80bbee84d" (UID: "08a13f88-6791-4bd9-a15f-55b80bbee84d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.019294 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.018651    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-envoy" pod "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" (UID: "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.019529 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.019504    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-podinfo" pod "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" (UID: "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:13:50.020192 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.020117    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-envoy" pod "e983d6ab-b49f-4e25-ba45-37f08b410917" (UID: "e983d6ab-b49f-4e25-ba45-37f08b410917"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.020192 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.020182    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e983d6ab-b49f-4e25-ba45-37f08b410917-kube-api-access-6zmsp" pod "e983d6ab-b49f-4e25-ba45-37f08b410917" (UID: "e983d6ab-b49f-4e25-ba45-37f08b410917"). InnerVolumeSpecName "kube-api-access-6zmsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:50.020824 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.020615    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-podinfo" pod "08a13f88-6791-4bd9-a15f-55b80bbee84d" (UID: "08a13f88-6791-4bd9-a15f-55b80bbee84d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:13:50.020824 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.020738    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a13f88-6791-4bd9-a15f-55b80bbee84d-kube-api-access-j5hrj" pod "08a13f88-6791-4bd9-a15f-55b80bbee84d" (UID: "08a13f88-6791-4bd9-a15f-55b80bbee84d"). InnerVolumeSpecName "kube-api-access-j5hrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:50.020824 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.020791    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-envoy" pod "08a13f88-6791-4bd9-a15f-55b80bbee84d" (UID: "08a13f88-6791-4bd9-a15f-55b80bbee84d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.021261 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.021237    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-token" pod "e983d6ab-b49f-4e25-ba45-37f08b410917" (UID: "e983d6ab-b49f-4e25-ba45-37f08b410917"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:50.021261 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.021135    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-token" pod "08a13f88-6791-4bd9-a15f-55b80bbee84d" (UID: "08a13f88-6791-4bd9-a15f-55b80bbee84d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:50.022054 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.022034    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-kube-api-access-zj5pn" pod "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" (UID: "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452"). InnerVolumeSpecName "kube-api-access-zj5pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:50.023050 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.023028    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-token" pod "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" (UID: "c5e00a1d-038e-4ed0-8a8c-a85ae64e5452"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:50.095487 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.095456    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:50.121908 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.121860    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5hrj\" (UniqueName: \"kubernetes.io/projected/08a13f88-6791-4bd9-a15f-55b80bbee84d-kube-api-access-j5hrj\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.121908 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.121908    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.121924    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.121936    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.121949    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.121966    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zmsp\" (UniqueName: \"kubernetes.io/projected/e983d6ab-b49f-4e25-ba45-37f08b410917-kube-api-access-6zmsp\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.121978    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj5pn\" (UniqueName: \"kubernetes.io/projected/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-kube-api-access-zj5pn\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.121990    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122002    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a13f88-6791-4bd9-a15f-55b80bbee84d-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122018    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122030    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122041    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122053    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122068    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122079    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122091    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122102    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122117    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122130    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122156    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122168    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122184    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122195    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122887 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122208    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e983d6ab-b49f-4e25-ba45-37f08b410917-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122887 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122219    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122887 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122234    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e983d6ab-b49f-4e25-ba45-37f08b410917-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.122887 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.122246    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a13f88-6791-4bd9-a15f-55b80bbee84d-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.554949 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.554916    4550 generic.go:356] "Generic (PLEG): container finished" podID="08a13f88-6791-4bd9-a15f-55b80bbee84d" containerID="a84b32722bf3b4ae4acb0ff4947c03a91f272f0ece6920561b1743f62625c755" exitCode=0
Oct 29 22:13:50.555370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.554976    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"
Oct 29 22:13:50.555370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.554998    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc" event={"ID":"08a13f88-6791-4bd9-a15f-55b80bbee84d","Type":"ContainerDied","Data":"a84b32722bf3b4ae4acb0ff4947c03a91f272f0ece6920561b1743f62625c755"}
Oct 29 22:13:50.555370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.555078    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc" event={"ID":"08a13f88-6791-4bd9-a15f-55b80bbee84d","Type":"ContainerDied","Data":"7d19890f36c0d0c3b4901094a96a0517464269b086f40dc8bb3705755234fc0c"}
Oct 29 22:13:50.555370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.555106    4550 scope.go:122] "RemoveContainer" containerID="a84b32722bf3b4ae4acb0ff4947c03a91f272f0ece6920561b1743f62625c755"
Oct 29 22:13:50.556830 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.556807    4550 generic.go:356] "Generic (PLEG): container finished" podID="e983d6ab-b49f-4e25-ba45-37f08b410917" containerID="dcb9ee2a52be56b6bd68d748ee4d7831861db6242b706ed549a9e2583ba60ffb" exitCode=0
Oct 29 22:13:50.556954 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.556840    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v" event={"ID":"e983d6ab-b49f-4e25-ba45-37f08b410917","Type":"ContainerDied","Data":"dcb9ee2a52be56b6bd68d748ee4d7831861db6242b706ed549a9e2583ba60ffb"}
Oct 29 22:13:50.556954 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.556892    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"
Oct 29 22:13:50.556954 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.556900    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v" event={"ID":"e983d6ab-b49f-4e25-ba45-37f08b410917","Type":"ContainerDied","Data":"4fd652f48b6f05e3fb5882f0aeed8fbd2e1b3c719039aca0396140cd2ccda4c5"}
Oct 29 22:13:50.558527 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.558506    4550 generic.go:356] "Generic (PLEG): container finished" podID="c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" containerID="508a7a1d9c3d6061d4939fd2d3ade7f41e619a6a4cf6c7293a8c81cb601c6288" exitCode=0
Oct 29 22:13:50.558639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.558567    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"
Oct 29 22:13:50.558639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.558575    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79" event={"ID":"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452","Type":"ContainerDied","Data":"508a7a1d9c3d6061d4939fd2d3ade7f41e619a6a4cf6c7293a8c81cb601c6288"}
Oct 29 22:13:50.558639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.558623    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79" event={"ID":"c5e00a1d-038e-4ed0-8a8c-a85ae64e5452","Type":"ContainerDied","Data":"e1cdd84e67725359b1aa19f8590ace372d94229de4293659bcbd0eac0e186e04"}
Oct 29 22:13:50.566201 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.566176    4550 scope.go:122] "RemoveContainer" containerID="a84b32722bf3b4ae4acb0ff4947c03a91f272f0ece6920561b1743f62625c755"
Oct 29 22:13:50.566543 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:13:50.566514    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a84b32722bf3b4ae4acb0ff4947c03a91f272f0ece6920561b1743f62625c755\": container with ID starting with a84b32722bf3b4ae4acb0ff4947c03a91f272f0ece6920561b1743f62625c755 not found: ID does not exist" containerID="a84b32722bf3b4ae4acb0ff4947c03a91f272f0ece6920561b1743f62625c755"
Oct 29 22:13:50.566666 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.566562    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a84b32722bf3b4ae4acb0ff4947c03a91f272f0ece6920561b1743f62625c755"} err="failed to get container status \"a84b32722bf3b4ae4acb0ff4947c03a91f272f0ece6920561b1743f62625c755\": rpc error: code = NotFound desc = could not find container \"a84b32722bf3b4ae4acb0ff4947c03a91f272f0ece6920561b1743f62625c755\": container with ID starting with a84b32722bf3b4ae4acb0ff4947c03a91f272f0ece6920561b1743f62625c755 not found: ID does not exist"
Oct 29 22:13:50.566666 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.566584    4550 scope.go:122] "RemoveContainer" containerID="dcb9ee2a52be56b6bd68d748ee4d7831861db6242b706ed549a9e2583ba60ffb"
Oct 29 22:13:50.578083 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.578063    4550 scope.go:122] "RemoveContainer" containerID="dcb9ee2a52be56b6bd68d748ee4d7831861db6242b706ed549a9e2583ba60ffb"
Oct 29 22:13:50.578391 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:13:50.578373    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb9ee2a52be56b6bd68d748ee4d7831861db6242b706ed549a9e2583ba60ffb\": container with ID starting with dcb9ee2a52be56b6bd68d748ee4d7831861db6242b706ed549a9e2583ba60ffb not found: ID does not exist" containerID="dcb9ee2a52be56b6bd68d748ee4d7831861db6242b706ed549a9e2583ba60ffb"
Oct 29 22:13:50.578447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.578400    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb9ee2a52be56b6bd68d748ee4d7831861db6242b706ed549a9e2583ba60ffb"} err="failed to get container status \"dcb9ee2a52be56b6bd68d748ee4d7831861db6242b706ed549a9e2583ba60ffb\": rpc error: code = NotFound desc = could not find container \"dcb9ee2a52be56b6bd68d748ee4d7831861db6242b706ed549a9e2583ba60ffb\": container with ID starting with dcb9ee2a52be56b6bd68d748ee4d7831861db6242b706ed549a9e2583ba60ffb not found: ID does not exist"
Oct 29 22:13:50.578447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.578418    4550 scope.go:122] "RemoveContainer" containerID="508a7a1d9c3d6061d4939fd2d3ade7f41e619a6a4cf6c7293a8c81cb601c6288"
Oct 29 22:13:50.587366 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.587352    4550 scope.go:122] "RemoveContainer" containerID="508a7a1d9c3d6061d4939fd2d3ade7f41e619a6a4cf6c7293a8c81cb601c6288"
Oct 29 22:13:50.587639 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:13:50.587618    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508a7a1d9c3d6061d4939fd2d3ade7f41e619a6a4cf6c7293a8c81cb601c6288\": container with ID starting with 508a7a1d9c3d6061d4939fd2d3ade7f41e619a6a4cf6c7293a8c81cb601c6288 not found: ID does not exist" containerID="508a7a1d9c3d6061d4939fd2d3ade7f41e619a6a4cf6c7293a8c81cb601c6288"
Oct 29 22:13:50.587707 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.587649    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508a7a1d9c3d6061d4939fd2d3ade7f41e619a6a4cf6c7293a8c81cb601c6288"} err="failed to get container status \"508a7a1d9c3d6061d4939fd2d3ade7f41e619a6a4cf6c7293a8c81cb601c6288\": rpc error: code = NotFound desc = could not find container \"508a7a1d9c3d6061d4939fd2d3ade7f41e619a6a4cf6c7293a8c81cb601c6288\": container with ID starting with 508a7a1d9c3d6061d4939fd2d3ade7f41e619a6a4cf6c7293a8c81cb601c6288 not found: ID does not exist"
Oct 29 22:13:50.620055 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.620013    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"]
Oct 29 22:13:50.631359 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.631335    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-fcq79"]
Oct 29 22:13:50.657415 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.657388    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"]
Oct 29 22:13:50.660893 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.660869    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-cj7rc"]
Oct 29 22:13:50.694638 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.694606    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"]
Oct 29 22:13:50.698054 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:50.698030    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-wmn2v"]
Oct 29 22:13:51.155533 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.155500    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:51.469241 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.469134    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main" containerID="cri-o://c74a06680102af036ec7a72f5e0b5442dc7967ff80d2fb5ff0c3f3ff06f161c9" gracePeriod=45
Oct 29 22:13:51.754097 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.754071    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:13:51.949016 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.948986    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660c3442-f3d1-4eb4-a013-232f82f8d81b-kube-api-access-b2bzq\" (UniqueName: \"kubernetes.io/projected/660c3442-f3d1-4eb4-a013-232f82f8d81b-kube-api-access-b2bzq\") pod \"660c3442-f3d1-4eb4-a013-232f82f8d81b\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") "
Oct 29 22:13:51.949233 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.949105    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-dshm\") pod \"660c3442-f3d1-4eb4-a013-232f82f8d81b\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") "
Oct 29 22:13:51.949320 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.949296    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-home\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-home\") pod \"660c3442-f3d1-4eb4-a013-232f82f8d81b\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") "
Oct 29 22:13:51.949377 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.949335    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/660c3442-f3d1-4eb4-a013-232f82f8d81b-tls-certs\" (UniqueName: \"kubernetes.io/secret/660c3442-f3d1-4eb4-a013-232f82f8d81b-tls-certs\") pod \"660c3442-f3d1-4eb4-a013-232f82f8d81b\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") "
Oct 29 22:13:51.949468 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.949449    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-model-cache\") pod \"660c3442-f3d1-4eb4-a013-232f82f8d81b\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") "
Oct 29 22:13:51.949540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.949512    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-tmp-dir\") pod \"660c3442-f3d1-4eb4-a013-232f82f8d81b\" (UID: \"660c3442-f3d1-4eb4-a013-232f82f8d81b\") "
Oct 29 22:13:51.949614 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.949540    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-home" pod "660c3442-f3d1-4eb4-a013-232f82f8d81b" (UID: "660c3442-f3d1-4eb4-a013-232f82f8d81b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:51.949757 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.949734    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-model-cache" pod "660c3442-f3d1-4eb4-a013-232f82f8d81b" (UID: "660c3442-f3d1-4eb4-a013-232f82f8d81b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:51.949885 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.949859    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-tmp-dir" pod "660c3442-f3d1-4eb4-a013-232f82f8d81b" (UID: "660c3442-f3d1-4eb4-a013-232f82f8d81b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:51.950130 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.950110    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-model-cache\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:51.950231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.950137    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-tmp-dir\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:51.950231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.950174    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-home\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:51.951953 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.951928    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660c3442-f3d1-4eb4-a013-232f82f8d81b-tls-certs" pod "660c3442-f3d1-4eb4-a013-232f82f8d81b" (UID: "660c3442-f3d1-4eb4-a013-232f82f8d81b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:51.951953 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.951932    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-dshm" pod "660c3442-f3d1-4eb4-a013-232f82f8d81b" (UID: "660c3442-f3d1-4eb4-a013-232f82f8d81b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:51.952110 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.952094    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660c3442-f3d1-4eb4-a013-232f82f8d81b-kube-api-access-b2bzq" pod "660c3442-f3d1-4eb4-a013-232f82f8d81b" (UID: "660c3442-f3d1-4eb4-a013-232f82f8d81b"). InnerVolumeSpecName "kube-api-access-b2bzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:51.991372 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.991302    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a13f88-6791-4bd9-a15f-55b80bbee84d" path="/var/lib/kubelet/pods/08a13f88-6791-4bd9-a15f-55b80bbee84d/volumes"
Oct 29 22:13:51.991853 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.991839    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" path="/var/lib/kubelet/pods/c5e00a1d-038e-4ed0-8a8c-a85ae64e5452/volumes"
Oct 29 22:13:51.992332 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:51.992317    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e983d6ab-b49f-4e25-ba45-37f08b410917" path="/var/lib/kubelet/pods/e983d6ab-b49f-4e25-ba45-37f08b410917/volumes"
Oct 29 22:13:52.064875 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:52.052313    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660c3442-f3d1-4eb4-a013-232f82f8d81b-tls-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.064875 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:52.052351    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2bzq\" (UniqueName: \"kubernetes.io/projected/660c3442-f3d1-4eb4-a013-232f82f8d81b-kube-api-access-b2bzq\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.064875 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:52.052365    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660c3442-f3d1-4eb4-a013-232f82f8d81b-dshm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.577307 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:52.577274    4550 generic.go:356] "Generic (PLEG): container finished" podID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerID="c74a06680102af036ec7a72f5e0b5442dc7967ff80d2fb5ff0c3f3ff06f161c9" exitCode=0
Oct 29 22:13:52.577513 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:52.577318    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"
Oct 29 22:13:52.577513 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:52.577338    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" event={"ID":"660c3442-f3d1-4eb4-a013-232f82f8d81b","Type":"ContainerDied","Data":"c74a06680102af036ec7a72f5e0b5442dc7967ff80d2fb5ff0c3f3ff06f161c9"}
Oct 29 22:13:52.577513 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:52.577376    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9" event={"ID":"660c3442-f3d1-4eb4-a013-232f82f8d81b","Type":"ContainerDied","Data":"57254b36415c8cfec25e58715d953ce903b7c2ddf7d1fd20fe06df5ec83f4ca9"}
Oct 29 22:13:52.577513 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:52.577396    4550 scope.go:122] "RemoveContainer" containerID="c74a06680102af036ec7a72f5e0b5442dc7967ff80d2fb5ff0c3f3ff06f161c9"
Oct 29 22:13:52.588053 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:52.588034    4550 scope.go:122] "RemoveContainer" containerID="c74a06680102af036ec7a72f5e0b5442dc7967ff80d2fb5ff0c3f3ff06f161c9"
Oct 29 22:13:52.588420 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:13:52.588386    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74a06680102af036ec7a72f5e0b5442dc7967ff80d2fb5ff0c3f3ff06f161c9\": container with ID starting with c74a06680102af036ec7a72f5e0b5442dc7967ff80d2fb5ff0c3f3ff06f161c9 not found: ID does not exist" containerID="c74a06680102af036ec7a72f5e0b5442dc7967ff80d2fb5ff0c3f3ff06f161c9"
Oct 29 22:13:52.588527 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:52.588432    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74a06680102af036ec7a72f5e0b5442dc7967ff80d2fb5ff0c3f3ff06f161c9"} err="failed to get container status \"c74a06680102af036ec7a72f5e0b5442dc7967ff80d2fb5ff0c3f3ff06f161c9\": rpc error: code = NotFound desc = could not find container \"c74a06680102af036ec7a72f5e0b5442dc7967ff80d2fb5ff0c3f3ff06f161c9\": container with ID starting with c74a06680102af036ec7a72f5e0b5442dc7967ff80d2fb5ff0c3f3ff06f161c9 not found: ID does not exist"
Oct 29 22:13:52.630773 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:52.630747    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"]
Oct 29 22:13:52.642893 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:52.642868    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/simulator-w4-b31f3f-kserve-7884846896-8npx9"]
Oct 29 22:13:52.863232 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:52.863121    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:53.004789 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:53.004757    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:53.126526 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:53.126450    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:53.339167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:53.338182    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:53.807040 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:53.807005    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:53.989988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:53.989956    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" path="/var/lib/kubelet/pods/660c3442-f3d1-4eb4-a013-232f82f8d81b/volumes"
Oct 29 22:13:55.183280 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:55.183245    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:56.104426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:56.104376    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:56.221617 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:56.221580    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:56.883215 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:56.883167    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:58.784528 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:58.784491    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"]
Oct 29 22:13:58.808121 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:58.801310    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"]
Oct 29 22:13:58.857724 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:58.857688    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:13:58.982044 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:13:58.982008    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:01.025825 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:01.025788    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:01.185032 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:01.184997    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:01.761171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:01.758497    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:02.204778 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:02.204737    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:02.256287 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:02.256251    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:14:02.256581 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:02.256549    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:02.257124 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:02.257097    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:14:02.277501 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:02.265218    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:14:02.363388 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:02.361235    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:02.502261 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:02.485423    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:02.765221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:02.761260    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:02.765221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:02.761836    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:14:02.965433 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:02.965390    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.242764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:03.209242    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.242764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:03.213164    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.242764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:03.214535    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:14:03.279327 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:03.279276    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:14:03.280081 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:03.280056    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:14:03.397191 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:03.382104    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.397191 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:03.382701    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:14:03.432195 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:03.432129    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.460866 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:03.460832    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:14:03.489297 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:03.489245    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.490270 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:03.490244    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:14:03.763090 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:03.762723    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.788393 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:03.782260    4550 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:14:03.788393 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:03.782343    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:04.268168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:04.247331    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:04.335173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:04.328429    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:14:04.359197 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:04.335532    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:04.407169 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:04.385220    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:04.408687 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:04.390984    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:14:04.460915 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:04.460861    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:14:04.540404 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:04.500310    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:14:04.643300 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:04.643259    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:14:04.647263 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:04.647218    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:04.805314 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:04.788203    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.250429 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:05.250377    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.255528 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:05.255490    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" containerID="cri-o://e6c233bb309b03bd1102aed7dda96acefac85e501b016c87a934c5dd2e7540eb" gracePeriod=45
Oct 29 22:14:05.331312 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:05.331260    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.345393 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:05.345344    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:14:05.427170 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:05.423767    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": read tcp 10.133.0.2:47028->10.133.0.51:8000: read: connection reset by peer"
Oct 29 22:14:05.503301 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:05.477092    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 29 22:14:05.503301 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:05.493181    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.503301 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:05.503265    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.584168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:05.583261    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:14:05.584168 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:05.583330    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.757681 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:05.757576    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": dial tcp 10.133.0.51:8000: connect: connection refused"
Oct 29 22:14:06.025707 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:06.025240    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-srchd" podUID="93b4187a-1551-4cf8-b807-d057baf03920" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:06.260204 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:06.260158    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:14:06.350351 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:06.350291    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:06.352429 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:06.351992    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:06.478378 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:06.478278    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:06.495305 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:06.495256    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:06.536904 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:06.536857    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:06.759645 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:06.759541    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": dial tcp 10.133.0.51:8000: connect: connection refused"
Oct 29 22:14:06.943927 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:06.943888    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:07.156280 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:07.139329    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" containerID="cri-o://66cbc33972c6cde7b16aa2fe38b8002cae3a26063ddba3f2daf6c8deafaca86c" gracePeriod=43
Oct 29 22:14:07.156744 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:07.148414    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:07.269276 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:07.269227    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:07.315362 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:07.315311    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": read tcp 10.133.0.2:47366->10.133.0.67:8000: read: connection reset by peer"
Oct 29 22:14:07.346629 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:07.346579    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": dial tcp 10.133.0.67:8000: connect: connection refused"
Oct 29 22:14:07.351248 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:07.351208    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:14:07.378166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:07.352216    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:07.481301 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:07.480268    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:07.562050 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:07.562003    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:07.761601 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:07.761504    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": dial tcp 10.133.0.51:8000: connect: connection refused"
Oct 29 22:14:08.216166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.210942    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:08.285494 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.274304    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:14:08.353778 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.353257    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": dial tcp 10.133.0.67:8000: connect: connection refused"
Oct 29 22:14:08.366104 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.361969    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:14:08.406198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.402728    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:08.487326 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.486246    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:08.519510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.518684    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:14:08.529421 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.526294    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:14:08.529421 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.527342    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:14:08.613635 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.570230    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:08.618399 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.618038    4550 generic.go:356] "Generic (PLEG): container finished" podID="cd727a21-2709-44be-a77d-f8cb1482e317" containerID="e6c233bb309b03bd1102aed7dda96acefac85e501b016c87a934c5dd2e7540eb" exitCode=0
Oct 29 22:14:08.618399 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.618249    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" event={"ID":"cd727a21-2709-44be-a77d-f8cb1482e317","Type":"ContainerDied","Data":"e6c233bb309b03bd1102aed7dda96acefac85e501b016c87a934c5dd2e7540eb"}
Oct 29 22:14:08.673805 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.671850    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:14:08.673971 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.673828    4550 generic.go:356] "Generic (PLEG): container finished" podID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerID="66cbc33972c6cde7b16aa2fe38b8002cae3a26063ddba3f2daf6c8deafaca86c" exitCode=0
Oct 29 22:14:08.673971 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.673874    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" event={"ID":"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0","Type":"ContainerDied","Data":"66cbc33972c6cde7b16aa2fe38b8002cae3a26063ddba3f2daf6c8deafaca86c"}
Oct 29 22:14:08.696456 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.695979    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:14:08.776763 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.776735    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:14:08.807134 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.807098    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-home\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-home\") pod \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") "
Oct 29 22:14:08.807325 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.807174    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-tmp-dir\") pod \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") "
Oct 29 22:14:08.807325 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.807217    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-dshm\") pod \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") "
Oct 29 22:14:08.807325 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.807281    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-kube-api-access-4p9kc\" (UniqueName: \"kubernetes.io/projected/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-kube-api-access-4p9kc\") pod \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") "
Oct 29 22:14:08.807498 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.807343    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-model-cache\") pod \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") "
Oct 29 22:14:08.807498 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.807455    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-tls-certs\") pod \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\" (UID: \"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0\") "
Oct 29 22:14:08.809288 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.809171    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-model-cache" pod "20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" (UID: "20bda0fc-d2ac-4435-9e5d-bc6b557e59b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:08.809500 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.809375    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-tmp-dir" pod "20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" (UID: "20bda0fc-d2ac-4435-9e5d-bc6b557e59b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:08.810832 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.810795    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-home" pod "20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" (UID: "20bda0fc-d2ac-4435-9e5d-bc6b557e59b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:08.811571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.811546    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-tls-certs" pod "20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" (UID: "20bda0fc-d2ac-4435-9e5d-bc6b557e59b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:08.813131 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.813106    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-dshm" pod "20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" (UID: "20bda0fc-d2ac-4435-9e5d-bc6b557e59b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:08.813673 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.813643    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-kube-api-access-4p9kc" pod "20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" (UID: "20bda0fc-d2ac-4435-9e5d-bc6b557e59b0"). InnerVolumeSpecName "kube-api-access-4p9kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:08.843519 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.843490    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:14:08.909387 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.909346    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-tmp-dir\") pod \"cd727a21-2709-44be-a77d-f8cb1482e317\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") "
Oct 29 22:14:08.909563 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.909443    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-dshm\") pod \"cd727a21-2709-44be-a77d-f8cb1482e317\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") "
Oct 29 22:14:08.909563 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.909501    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd727a21-2709-44be-a77d-f8cb1482e317-kube-api-access-xvhwf\" (UniqueName: \"kubernetes.io/projected/cd727a21-2709-44be-a77d-f8cb1482e317-kube-api-access-xvhwf\") pod \"cd727a21-2709-44be-a77d-f8cb1482e317\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") "
Oct 29 22:14:08.909563 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.909555    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-model-cache\") pod \"cd727a21-2709-44be-a77d-f8cb1482e317\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") "
Oct 29 22:14:08.909891 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.909653    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd727a21-2709-44be-a77d-f8cb1482e317-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd727a21-2709-44be-a77d-f8cb1482e317-tls-certs\") pod \"cd727a21-2709-44be-a77d-f8cb1482e317\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") "
Oct 29 22:14:08.909891 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.909665    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-tmp-dir" pod "cd727a21-2709-44be-a77d-f8cb1482e317" (UID: "cd727a21-2709-44be-a77d-f8cb1482e317"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:08.909891 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.909706    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-home\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-home\") pod \"cd727a21-2709-44be-a77d-f8cb1482e317\" (UID: \"cd727a21-2709-44be-a77d-f8cb1482e317\") "
Oct 29 22:14:08.910378 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.910338    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-home" pod "cd727a21-2709-44be-a77d-f8cb1482e317" (UID: "cd727a21-2709-44be-a77d-f8cb1482e317"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:08.910378 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.910354    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-home\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:08.910378 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.910376    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-tmp-dir\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:08.910641 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.910390    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-dshm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:08.910641 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.910405    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p9kc\" (UniqueName: \"kubernetes.io/projected/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-kube-api-access-4p9kc\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:08.910641 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.910419    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-model-cache\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:08.910641 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.910434    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-tmp-dir\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:08.910641 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.910450    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0-tls-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:08.910641 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.910600    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-model-cache" pod "cd727a21-2709-44be-a77d-f8cb1482e317" (UID: "cd727a21-2709-44be-a77d-f8cb1482e317"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:08.913329 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.913305    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd727a21-2709-44be-a77d-f8cb1482e317-kube-api-access-xvhwf" pod "cd727a21-2709-44be-a77d-f8cb1482e317" (UID: "cd727a21-2709-44be-a77d-f8cb1482e317"). InnerVolumeSpecName "kube-api-access-xvhwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:08.913438 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.913367    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-dshm" pod "cd727a21-2709-44be-a77d-f8cb1482e317" (UID: "cd727a21-2709-44be-a77d-f8cb1482e317"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:08.914601 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:08.914577    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd727a21-2709-44be-a77d-f8cb1482e317-tls-certs" pod "cd727a21-2709-44be-a77d-f8cb1482e317" (UID: "cd727a21-2709-44be-a77d-f8cb1482e317"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:09.011891 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.011809    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd727a21-2709-44be-a77d-f8cb1482e317-tls-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:09.011891 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.011843    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-home\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:09.011891 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.011851    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-dshm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:09.011891 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.011860    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvhwf\" (UniqueName: \"kubernetes.io/projected/cd727a21-2709-44be-a77d-f8cb1482e317-kube-api-access-xvhwf\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:09.011891 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.011869    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd727a21-2709-44be-a77d-f8cb1482e317-model-cache\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:09.681478 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.681447    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"
Oct 29 22:14:09.681941 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.681473    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb" event={"ID":"20bda0fc-d2ac-4435-9e5d-bc6b557e59b0","Type":"ContainerDied","Data":"0869884d3755931bd130a5e0a878fb66dcaa6e6c6e92b318fc7a2fbcbfe51b91"}
Oct 29 22:14:09.681941 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.681526    4550 scope.go:122] "RemoveContainer" containerID="66cbc33972c6cde7b16aa2fe38b8002cae3a26063ddba3f2daf6c8deafaca86c"
Oct 29 22:14:09.683011 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.682994    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" event={"ID":"cd727a21-2709-44be-a77d-f8cb1482e317","Type":"ContainerDied","Data":"7e1c2349f02ae92beb37a4bbb3966677cd85de9fbc532c2aca5b71dc445db7a6"}
Oct 29 22:14:09.683084 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.683016    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"
Oct 29 22:14:09.691953 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.691929    4550 scope.go:122] "RemoveContainer" containerID="e6c233bb309b03bd1102aed7dda96acefac85e501b016c87a934c5dd2e7540eb"
Oct 29 22:14:09.734854 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.734819    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"]
Oct 29 22:14:09.745948 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.745921    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/distinct-2-w6-a940bc-kserve-5ff54cdc7-cltkb"]
Oct 29 22:14:09.762252 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.762218    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:09.778875 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.778838    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"]
Oct 29 22:14:09.781922 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.781902    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/premium-w6-a940bc-kserve-75f5c84cdc-s7jk8"]
Oct 29 22:14:09.990302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.990228    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" path="/var/lib/kubelet/pods/20bda0fc-d2ac-4435-9e5d-bc6b557e59b0/volumes"
Oct 29 22:14:09.990718 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:09.990704    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" path="/var/lib/kubelet/pods/cd727a21-2709-44be-a77d-f8cb1482e317/volumes"
Oct 29 22:14:10.480037 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:10.479999    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:10.661190 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:10.661159    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:10.800607 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:10.800519    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:10.999173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:10.999103    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:11.199237 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:11.199203    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:11.491409 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:11.491319    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:14:11.676951 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:11.676918    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:12.919378 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:12.919347    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:13.429403 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:13.429352    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:13.834915 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:13.834870    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:13.858719 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:13.857301    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" containerID="cri-o://f51df323877787bc7c33895e982d70894d2ce0dd8f694ce45547a5d5ebda9993" gracePeriod=45
Oct 29 22:14:13.940174 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:13.937506    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" containerID="cri-o://5737bb8f15a270c47dfcf47588883fc8bb33ab86e947b008275217751d4bb0fc" gracePeriod=45
Oct 29 22:14:14.203974 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.203942    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:14.386224 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.382055    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:14.549995 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.549969    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:14:14.555072 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.555046    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:14:14.573862 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.573835    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:14.688606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.684306    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-model-cache\") pod \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") "
Oct 29 22:14:14.688606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.684459    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-home\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-home\") pod \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") "
Oct 29 22:14:14.688606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.684494    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-dshm\") pod \"09bace6e-b116-4c71-a2c1-1265a83b1608\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") "
Oct 29 22:14:14.688606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.684545    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-tmp-dir\") pod \"09bace6e-b116-4c71-a2c1-1265a83b1608\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") "
Oct 29 22:14:14.688606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.684610    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-kube-api-access-xj9p2\" (UniqueName: \"kubernetes.io/projected/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-kube-api-access-xj9p2\") pod \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") "
Oct 29 22:14:14.688606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.684639    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-model-cache\") pod \"09bace6e-b116-4c71-a2c1-1265a83b1608\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") "
Oct 29 22:14:14.688606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.684686    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09bace6e-b116-4c71-a2c1-1265a83b1608-tls-certs\" (UniqueName: \"kubernetes.io/secret/09bace6e-b116-4c71-a2c1-1265a83b1608-tls-certs\") pod \"09bace6e-b116-4c71-a2c1-1265a83b1608\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") "
Oct 29 22:14:14.688606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.684781    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09bace6e-b116-4c71-a2c1-1265a83b1608-kube-api-access-8g8kc\" (UniqueName: \"kubernetes.io/projected/09bace6e-b116-4c71-a2c1-1265a83b1608-kube-api-access-8g8kc\") pod \"09bace6e-b116-4c71-a2c1-1265a83b1608\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") "
Oct 29 22:14:14.688606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.684837    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-tmp-dir\") pod \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") "
Oct 29 22:14:14.688606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.684905    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-dshm\") pod \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") "
Oct 29 22:14:14.688606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.684999    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-home\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-home\") pod \"09bace6e-b116-4c71-a2c1-1265a83b1608\" (UID: \"09bace6e-b116-4c71-a2c1-1265a83b1608\") "
Oct 29 22:14:14.688606 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.685057    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-tls-certs\") pod \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\" (UID: \"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd\") "
Oct 29 22:14:14.707999 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.707726    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-tmp-dir" pod "09bace6e-b116-4c71-a2c1-1265a83b1608" (UID: "09bace6e-b116-4c71-a2c1-1265a83b1608"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:14.708438 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.708405    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-model-cache" pod "09bace6e-b116-4c71-a2c1-1265a83b1608" (UID: "09bace6e-b116-4c71-a2c1-1265a83b1608"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:14.709037 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.708994    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-model-cache" pod "66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" (UID: "66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:14.710599 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.710302    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-dshm" pod "09bace6e-b116-4c71-a2c1-1265a83b1608" (UID: "09bace6e-b116-4c71-a2c1-1265a83b1608"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:14.712560 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.711480    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-dshm" pod "66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" (UID: "66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:14.712560 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.711733    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-tmp-dir" pod "66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" (UID: "66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:14.712560 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.711920    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-home" pod "09bace6e-b116-4c71-a2c1-1265a83b1608" (UID: "09bace6e-b116-4c71-a2c1-1265a83b1608"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:14.712560 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.712300    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-home" pod "66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" (UID: "66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:14.713205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.713169    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09bace6e-b116-4c71-a2c1-1265a83b1608-tls-certs" pod "09bace6e-b116-4c71-a2c1-1265a83b1608" (UID: "09bace6e-b116-4c71-a2c1-1265a83b1608"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:14.757466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.727115    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-tls-certs" pod "66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" (UID: "66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:14.757466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.737461    4550 generic.go:356] "Generic (PLEG): container finished" podID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerID="5737bb8f15a270c47dfcf47588883fc8bb33ab86e947b008275217751d4bb0fc" exitCode=0
Oct 29 22:14:14.757466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.737565    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" event={"ID":"09bace6e-b116-4c71-a2c1-1265a83b1608","Type":"ContainerDied","Data":"5737bb8f15a270c47dfcf47588883fc8bb33ab86e947b008275217751d4bb0fc"}
Oct 29 22:14:14.757466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.737609    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" event={"ID":"09bace6e-b116-4c71-a2c1-1265a83b1608","Type":"ContainerDied","Data":"4f25640d63cba2ea1a06a65aa2c1f771782af08a3f2e3879ffc2a397c975e61e"}
Oct 29 22:14:14.757466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.737633    4550 scope.go:122] "RemoveContainer" containerID="5737bb8f15a270c47dfcf47588883fc8bb33ab86e947b008275217751d4bb0fc"
Oct 29 22:14:14.757466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.738335    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"
Oct 29 22:14:14.757466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.740633    4550 generic.go:356] "Generic (PLEG): container finished" podID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerID="f51df323877787bc7c33895e982d70894d2ce0dd8f694ce45547a5d5ebda9993" exitCode=0
Oct 29 22:14:14.757466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.740855    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" event={"ID":"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd","Type":"ContainerDied","Data":"f51df323877787bc7c33895e982d70894d2ce0dd8f694ce45547a5d5ebda9993"}
Oct 29 22:14:14.757466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.740890    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" event={"ID":"66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd","Type":"ContainerDied","Data":"ff79c1799bd2704d69ba8a0d86c37639fec189c15cee154b01079948f1f27cce"}
Oct 29 22:14:14.757466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.740976    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"
Oct 29 22:14:14.758029 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.757946    4550 scope.go:122] "RemoveContainer" containerID="5737bb8f15a270c47dfcf47588883fc8bb33ab86e947b008275217751d4bb0fc"
Oct 29 22:14:14.758568 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:14.758539    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5737bb8f15a270c47dfcf47588883fc8bb33ab86e947b008275217751d4bb0fc\": container with ID starting with 5737bb8f15a270c47dfcf47588883fc8bb33ab86e947b008275217751d4bb0fc not found: ID does not exist" containerID="5737bb8f15a270c47dfcf47588883fc8bb33ab86e947b008275217751d4bb0fc"
Oct 29 22:14:14.758762 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.758737    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5737bb8f15a270c47dfcf47588883fc8bb33ab86e947b008275217751d4bb0fc"} err="failed to get container status \"5737bb8f15a270c47dfcf47588883fc8bb33ab86e947b008275217751d4bb0fc\": rpc error: code = NotFound desc = could not find container \"5737bb8f15a270c47dfcf47588883fc8bb33ab86e947b008275217751d4bb0fc\": container with ID starting with 5737bb8f15a270c47dfcf47588883fc8bb33ab86e947b008275217751d4bb0fc not found: ID does not exist"
Oct 29 22:14:14.758922 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.758869    4550 scope.go:122] "RemoveContainer" containerID="f51df323877787bc7c33895e982d70894d2ce0dd8f694ce45547a5d5ebda9993"
Oct 29 22:14:14.789188 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.767682    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bace6e-b116-4c71-a2c1-1265a83b1608-kube-api-access-8g8kc" pod "09bace6e-b116-4c71-a2c1-1265a83b1608" (UID: "09bace6e-b116-4c71-a2c1-1265a83b1608"). InnerVolumeSpecName "kube-api-access-8g8kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:14.801240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.789446    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-home\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:14.801240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.789479    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-dshm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:14.801240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.789523    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-tmp-dir\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:14.801240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.789538    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-model-cache\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:14.801240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.789560    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09bace6e-b116-4c71-a2c1-1265a83b1608-tls-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:14.801240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.789574    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g8kc\" (UniqueName: \"kubernetes.io/projected/09bace6e-b116-4c71-a2c1-1265a83b1608-kube-api-access-8g8kc\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:14.801240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.789589    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-tmp-dir\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:14.801240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.789603    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-dshm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:14.801240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.789623    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09bace6e-b116-4c71-a2c1-1265a83b1608-home\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:14.801240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.789637    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-tls-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:14.801240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.789651    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-model-cache\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:14.803079 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.802879    4550 scope.go:122] "RemoveContainer" containerID="f51df323877787bc7c33895e982d70894d2ce0dd8f694ce45547a5d5ebda9993"
Oct 29 22:14:14.803422 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.803387    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-kube-api-access-xj9p2" pod "66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" (UID: "66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd"). InnerVolumeSpecName "kube-api-access-xj9p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:14.805176 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:14.805034    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51df323877787bc7c33895e982d70894d2ce0dd8f694ce45547a5d5ebda9993\": container with ID starting with f51df323877787bc7c33895e982d70894d2ce0dd8f694ce45547a5d5ebda9993 not found: ID does not exist" containerID="f51df323877787bc7c33895e982d70894d2ce0dd8f694ce45547a5d5ebda9993"
Oct 29 22:14:14.805176 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.805087    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51df323877787bc7c33895e982d70894d2ce0dd8f694ce45547a5d5ebda9993"} err="failed to get container status \"f51df323877787bc7c33895e982d70894d2ce0dd8f694ce45547a5d5ebda9993\": rpc error: code = NotFound desc = could not find container \"f51df323877787bc7c33895e982d70894d2ce0dd8f694ce45547a5d5ebda9993\": container with ID starting with f51df323877787bc7c33895e982d70894d2ce0dd8f694ce45547a5d5ebda9993 not found: ID does not exist"
Oct 29 22:14:14.891184 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:14.891135    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj9p2\" (UniqueName: \"kubernetes.io/projected/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd-kube-api-access-xj9p2\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:15.095572 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:15.095537    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"]
Oct 29 22:14:15.101763 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:15.101726    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g"]
Oct 29 22:14:15.165726 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:15.165688    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"]
Oct 29 22:14:15.174255 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:15.174215    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r"]
Oct 29 22:14:15.229930 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:15.229890    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:15.255537 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:15.255495    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/simulator-w5-743a5e-kserve-796d7465f-c889r" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:15.373844 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:15.373794    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-w5-743a5e-kserve-6c6d6c9585-nd29g" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:15.592908 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:15.592866    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:15.989435 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:15.989400    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" path="/var/lib/kubelet/pods/09bace6e-b116-4c71-a2c1-1265a83b1608/volumes"
Oct 29 22:14:15.989904 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:15.989891    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" path="/var/lib/kubelet/pods/66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd/volumes"
Oct 29 22:14:17.155533 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:17.155501    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:17.749811 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:17.749774    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:17.913021 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:17.912913    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:18.374373 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:18.374322    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:18.677488 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:18.677408    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:18.795632 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:18.795594    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:19.074631 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:19.074604    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:20.104961 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:20.104921    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:22.277635 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:22.275031    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:22.424997 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:22.424964    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:22.730856 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:22.730820    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:24.947060 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:24.947029    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:25.084750 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:25.084566    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:25.284459 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:25.284381    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"]
Oct 29 22:14:25.284631 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:25.284588    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h" podUID="be3e55fd-83b9-4c52-a2a9-28da7d04477e" containerName="istio-proxy" containerID="cri-o://1dfd9bcf245f00955db5eb0a1fb055e5845e44aee42dc1e805567c8e4ebb0956" gracePeriod=30
Oct 29 22:14:25.300269 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:25.300110    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"]
Oct 29 22:14:25.300463 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:25.300414    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk" podUID="2913dd75-7d51-4ee4-b705-f6a7396d75de" containerName="istio-proxy" containerID="cri-o://c690d85e84e3d7874dc412f661d0a0128754f47b29e1d323b07d9852b3881237" gracePeriod=30
Oct 29 22:14:25.308110 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:25.308079    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"]
Oct 29 22:14:25.308336 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:25.308313    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerName="istio-proxy" containerID="cri-o://33c3422c262f448be1cfdf14a0d935e658d0728f5914a97e33160f899c39d4a6" gracePeriod=30
Oct 29 22:14:25.317506 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:25.317481    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:25.515370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:25.515332    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:25.516915 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:25.516878    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h" podUID="be3e55fd-83b9-4c52-a2a9-28da7d04477e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:25.750786 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:25.750751    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:25.750786 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:25.750827    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:26.767848 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:26.767804    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:28.244317 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:28.244255    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:28.369615 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:28.369571    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:28.565757 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:28.565635    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:28.894542 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:28.894509    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:29.022440 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:29.022340    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:29.178859 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:29.178782    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:29.520643 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:29.520569    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:29.613964 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:29.613934    4550 scope.go:122] "RemoveContainer" containerID="e7f9be395d8e152ddb2fbbcedcaacd28d383fc7ab90c8331751d147d2d6b98b5"
Oct 29 22:14:30.620288 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.611564    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 22:14:30.676866 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.676837    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 22:14:30.704028 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.703994    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:14:30.716991 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.716950    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istiod-ca-cert\") pod \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") "
Oct 29 22:14:30.717194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717006    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-workload-socket\") pod \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") "
Oct 29 22:14:30.717194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717089    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-workload-socket\") pod \"2913dd75-7d51-4ee4-b705-f6a7396d75de\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") "
Oct 29 22:14:30.717194 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717125    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3e55fd-83b9-4c52-a2a9-28da7d04477e-kube-api-access-55zwm\" (UniqueName: \"kubernetes.io/projected/be3e55fd-83b9-4c52-a2a9-28da7d04477e-kube-api-access-55zwm\") pod \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") "
Oct 29 22:14:30.717426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717267    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-podinfo\") pod \"2913dd75-7d51-4ee4-b705-f6a7396d75de\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") "
Oct 29 22:14:30.717426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717301    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-token\" (UniqueName: \"kubernetes.io/projected/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-token\") pod \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") "
Oct 29 22:14:30.717426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717341    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2913dd75-7d51-4ee4-b705-f6a7396d75de-kube-api-access-c9rbr\" (UniqueName: \"kubernetes.io/projected/2913dd75-7d51-4ee4-b705-f6a7396d75de-kube-api-access-c9rbr\") pod \"2913dd75-7d51-4ee4-b705-f6a7396d75de\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") "
Oct 29 22:14:30.717426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717387    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istiod-ca-cert" pod "be3e55fd-83b9-4c52-a2a9-28da7d04477e" (UID: "be3e55fd-83b9-4c52-a2a9-28da7d04477e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:30.717426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717401    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-podinfo\") pod \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") "
Oct 29 22:14:30.717692 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717424    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-workload-socket" pod "2913dd75-7d51-4ee4-b705-f6a7396d75de" (UID: "2913dd75-7d51-4ee4-b705-f6a7396d75de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.717692 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717452    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-envoy\") pod \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") "
Oct 29 22:14:30.717692 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717487    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-credential-socket\") pod \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") "
Oct 29 22:14:30.717692 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717564    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-credential-socket\") pod \"2913dd75-7d51-4ee4-b705-f6a7396d75de\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") "
Oct 29 22:14:30.717692 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717623    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2913dd75-7d51-4ee4-b705-f6a7396d75de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2913dd75-7d51-4ee4-b705-f6a7396d75de-istiod-ca-cert\") pod \"2913dd75-7d51-4ee4-b705-f6a7396d75de\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") "
Oct 29 22:14:30.717692 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717668    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-data\") pod \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") "
Oct 29 22:14:30.718013 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717741    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-workload-certs\") pod \"2913dd75-7d51-4ee4-b705-f6a7396d75de\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") "
Oct 29 22:14:30.718013 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717769    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-workload-certs\") pod \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\" (UID: \"be3e55fd-83b9-4c52-a2a9-28da7d04477e\") "
Oct 29 22:14:30.718013 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717829    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-data\") pod \"2913dd75-7d51-4ee4-b705-f6a7396d75de\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") "
Oct 29 22:14:30.718013 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717872    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-token\" (UniqueName: \"kubernetes.io/projected/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-token\") pod \"2913dd75-7d51-4ee4-b705-f6a7396d75de\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") "
Oct 29 22:14:30.718013 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717873    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-credential-socket" pod "be3e55fd-83b9-4c52-a2a9-28da7d04477e" (UID: "be3e55fd-83b9-4c52-a2a9-28da7d04477e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.718013 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.717915    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-envoy\") pod \"2913dd75-7d51-4ee4-b705-f6a7396d75de\" (UID: \"2913dd75-7d51-4ee4-b705-f6a7396d75de\") "
Oct 29 22:14:30.718617 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.718586    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-workload-socket" pod "be3e55fd-83b9-4c52-a2a9-28da7d04477e" (UID: "be3e55fd-83b9-4c52-a2a9-28da7d04477e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.718617 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.718603    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.718788 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.718626    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.718788 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.718643    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.718903 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.718841    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-workload-certs" pod "2913dd75-7d51-4ee4-b705-f6a7396d75de" (UID: "2913dd75-7d51-4ee4-b705-f6a7396d75de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.719061 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.719039    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-credential-socket" pod "2913dd75-7d51-4ee4-b705-f6a7396d75de" (UID: "2913dd75-7d51-4ee4-b705-f6a7396d75de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.719289 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.719174    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-data" pod "be3e55fd-83b9-4c52-a2a9-28da7d04477e" (UID: "be3e55fd-83b9-4c52-a2a9-28da7d04477e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.719289 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.719269    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-data" pod "2913dd75-7d51-4ee4-b705-f6a7396d75de" (UID: "2913dd75-7d51-4ee4-b705-f6a7396d75de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.719481 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.719398    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2913dd75-7d51-4ee4-b705-f6a7396d75de-istiod-ca-cert" pod "2913dd75-7d51-4ee4-b705-f6a7396d75de" (UID: "2913dd75-7d51-4ee4-b705-f6a7396d75de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:30.719664 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.719629    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-workload-certs" pod "be3e55fd-83b9-4c52-a2a9-28da7d04477e" (UID: "be3e55fd-83b9-4c52-a2a9-28da7d04477e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.721264 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.721240    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-podinfo" pod "be3e55fd-83b9-4c52-a2a9-28da7d04477e" (UID: "be3e55fd-83b9-4c52-a2a9-28da7d04477e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:30.721370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.721259    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-token" pod "be3e55fd-83b9-4c52-a2a9-28da7d04477e" (UID: "be3e55fd-83b9-4c52-a2a9-28da7d04477e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.721614 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.721587    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-envoy" pod "be3e55fd-83b9-4c52-a2a9-28da7d04477e" (UID: "be3e55fd-83b9-4c52-a2a9-28da7d04477e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.722116 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.722098    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-envoy" pod "2913dd75-7d51-4ee4-b705-f6a7396d75de" (UID: "2913dd75-7d51-4ee4-b705-f6a7396d75de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.722253 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.722231    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-token" pod "2913dd75-7d51-4ee4-b705-f6a7396d75de" (UID: "2913dd75-7d51-4ee4-b705-f6a7396d75de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.723783 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.723748    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2913dd75-7d51-4ee4-b705-f6a7396d75de-kube-api-access-c9rbr" pod "2913dd75-7d51-4ee4-b705-f6a7396d75de" (UID: "2913dd75-7d51-4ee4-b705-f6a7396d75de"). InnerVolumeSpecName "kube-api-access-c9rbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.724113 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.724079    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-podinfo" pod "2913dd75-7d51-4ee4-b705-f6a7396d75de" (UID: "2913dd75-7d51-4ee4-b705-f6a7396d75de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:30.724387 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.724361    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3e55fd-83b9-4c52-a2a9-28da7d04477e-kube-api-access-55zwm" pod "be3e55fd-83b9-4c52-a2a9-28da7d04477e" (UID: "be3e55fd-83b9-4c52-a2a9-28da7d04477e"). InnerVolumeSpecName "kube-api-access-55zwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.819187 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.819151    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-data\") pod \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") "
Oct 29 22:14:30.819387 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.819237    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-credential-socket\") pod \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") "
Oct 29 22:14:30.819387 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.819294    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-envoy\") pod \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") "
Oct 29 22:14:30.819544 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.819421    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-workload-socket\") pod \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") "
Oct 29 22:14:30.819544 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.819487    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-workload-certs\") pod \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") "
Oct 29 22:14:30.819544 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.819506    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-data" pod "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" (UID: "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.819544 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.819526    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-kube-api-access-f9xd2\" (UniqueName: \"kubernetes.io/projected/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-kube-api-access-f9xd2\") pod \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") "
Oct 29 22:14:30.819769 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.819544    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-credential-socket" pod "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" (UID: "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.819769 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.819677    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istiod-ca-cert\") pod \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") "
Oct 29 22:14:30.819769 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.819717    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-token\" (UniqueName: \"kubernetes.io/projected/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-token\") pod \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") "
Oct 29 22:14:30.819769 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.819749    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-podinfo\") pod \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\" (UID: \"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9\") "
Oct 29 22:14:30.820170 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820045    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istiod-ca-cert" pod "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" (UID: "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:30.820422 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820397    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-workload-certs" pod "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" (UID: "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.820541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820450    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-workload-socket" pod "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" (UID: "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.820541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820487    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.820541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820515    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.820541 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820528    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.821003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820544    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55zwm\" (UniqueName: \"kubernetes.io/projected/be3e55fd-83b9-4c52-a2a9-28da7d04477e-kube-api-access-55zwm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.821003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820559    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.821003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820571    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.821003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820583    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9rbr\" (UniqueName: \"kubernetes.io/projected/2913dd75-7d51-4ee4-b705-f6a7396d75de-kube-api-access-c9rbr\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.821003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820596    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.821003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820611    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.821003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820622    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.821003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820634    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2913dd75-7d51-4ee4-b705-f6a7396d75de-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.821003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820657    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.821003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820669    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.821003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820682    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3e55fd-83b9-4c52-a2a9-28da7d04477e-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.821003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820694    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.821003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820706    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.821003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820718    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.821003 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.820730    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2913dd75-7d51-4ee4-b705-f6a7396d75de-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.822301 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.822276    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-kube-api-access-f9xd2" pod "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" (UID: "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9"). InnerVolumeSpecName "kube-api-access-f9xd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.822301 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.822289    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-token" pod "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" (UID: "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.822530 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.822515    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-podinfo" pod "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" (UID: "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:30.823363 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.823340    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-envoy" pod "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" (UID: "e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.907369 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.907335    4550 generic.go:356] "Generic (PLEG): container finished" podID="be3e55fd-83b9-4c52-a2a9-28da7d04477e" containerID="1dfd9bcf245f00955db5eb0a1fb055e5845e44aee42dc1e805567c8e4ebb0956" exitCode=0
Oct 29 22:14:30.907520 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.907390    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"
Oct 29 22:14:30.907520 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.907421    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h" event={"ID":"be3e55fd-83b9-4c52-a2a9-28da7d04477e","Type":"ContainerDied","Data":"1dfd9bcf245f00955db5eb0a1fb055e5845e44aee42dc1e805567c8e4ebb0956"}
Oct 29 22:14:30.907520 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.907477    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h" event={"ID":"be3e55fd-83b9-4c52-a2a9-28da7d04477e","Type":"ContainerDied","Data":"1ee2f4b4f7b5092a3b10677ad97be15367861bb93fc793ca08cbb1129e01466f"}
Oct 29 22:14:30.907520 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.907505    4550 scope.go:122] "RemoveContainer" containerID="1dfd9bcf245f00955db5eb0a1fb055e5845e44aee42dc1e805567c8e4ebb0956"
Oct 29 22:14:30.908939 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.908916    4550 generic.go:356] "Generic (PLEG): container finished" podID="2913dd75-7d51-4ee4-b705-f6a7396d75de" containerID="c690d85e84e3d7874dc412f661d0a0128754f47b29e1d323b07d9852b3881237" exitCode=0
Oct 29 22:14:30.909055 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.908946    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk" event={"ID":"2913dd75-7d51-4ee4-b705-f6a7396d75de","Type":"ContainerDied","Data":"c690d85e84e3d7874dc412f661d0a0128754f47b29e1d323b07d9852b3881237"}
Oct 29 22:14:30.909055 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.908966    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"
Oct 29 22:14:30.909055 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.909017    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk" event={"ID":"2913dd75-7d51-4ee4-b705-f6a7396d75de","Type":"ContainerDied","Data":"e4dff8ec1434625f0ff1a675088113513e6b62dc2061b3047686a70ae9f8f233"}
Oct 29 22:14:30.911446 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.911420    4550 generic.go:356] "Generic (PLEG): container finished" podID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerID="33c3422c262f448be1cfdf14a0d935e658d0728f5914a97e33160f899c39d4a6" exitCode=0
Oct 29 22:14:30.911577 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.911500    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"
Oct 29 22:14:30.911577 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.911560    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" event={"ID":"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9","Type":"ContainerDied","Data":"33c3422c262f448be1cfdf14a0d935e658d0728f5914a97e33160f899c39d4a6"}
Oct 29 22:14:30.911694 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.911602    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7" event={"ID":"e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9","Type":"ContainerDied","Data":"8b1d0d86b63bd65c44367eebd32fd3a5a010e7ee75d157b99ab263f67da31ce7"}
Oct 29 22:14:30.920272 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.920257    4550 scope.go:122] "RemoveContainer" containerID="1dfd9bcf245f00955db5eb0a1fb055e5845e44aee42dc1e805567c8e4ebb0956"
Oct 29 22:14:30.920534 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:30.920506    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfd9bcf245f00955db5eb0a1fb055e5845e44aee42dc1e805567c8e4ebb0956\": container with ID starting with 1dfd9bcf245f00955db5eb0a1fb055e5845e44aee42dc1e805567c8e4ebb0956 not found: ID does not exist" containerID="1dfd9bcf245f00955db5eb0a1fb055e5845e44aee42dc1e805567c8e4ebb0956"
Oct 29 22:14:30.920633 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.920548    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfd9bcf245f00955db5eb0a1fb055e5845e44aee42dc1e805567c8e4ebb0956"} err="failed to get container status \"1dfd9bcf245f00955db5eb0a1fb055e5845e44aee42dc1e805567c8e4ebb0956\": rpc error: code = NotFound desc = could not find container \"1dfd9bcf245f00955db5eb0a1fb055e5845e44aee42dc1e805567c8e4ebb0956\": container with ID starting with 1dfd9bcf245f00955db5eb0a1fb055e5845e44aee42dc1e805567c8e4ebb0956 not found: ID does not exist"
Oct 29 22:14:30.920633 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.920574    4550 scope.go:122] "RemoveContainer" containerID="c690d85e84e3d7874dc412f661d0a0128754f47b29e1d323b07d9852b3881237"
Oct 29 22:14:30.922033 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.922013    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.922112 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.922036    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.922112 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.922052    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.922112 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.922066    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9xd2\" (UniqueName: \"kubernetes.io/projected/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-kube-api-access-f9xd2\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.922112 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.922078    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.922112 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.922090    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.929803 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.929785    4550 scope.go:122] "RemoveContainer" containerID="c690d85e84e3d7874dc412f661d0a0128754f47b29e1d323b07d9852b3881237"
Oct 29 22:14:30.930061 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:30.930031    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c690d85e84e3d7874dc412f661d0a0128754f47b29e1d323b07d9852b3881237\": container with ID starting with c690d85e84e3d7874dc412f661d0a0128754f47b29e1d323b07d9852b3881237 not found: ID does not exist" containerID="c690d85e84e3d7874dc412f661d0a0128754f47b29e1d323b07d9852b3881237"
Oct 29 22:14:30.930131 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.930057    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c690d85e84e3d7874dc412f661d0a0128754f47b29e1d323b07d9852b3881237"} err="failed to get container status \"c690d85e84e3d7874dc412f661d0a0128754f47b29e1d323b07d9852b3881237\": rpc error: code = NotFound desc = could not find container \"c690d85e84e3d7874dc412f661d0a0128754f47b29e1d323b07d9852b3881237\": container with ID starting with c690d85e84e3d7874dc412f661d0a0128754f47b29e1d323b07d9852b3881237 not found: ID does not exist"
Oct 29 22:14:30.930131 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.930073    4550 scope.go:122] "RemoveContainer" containerID="33c3422c262f448be1cfdf14a0d935e658d0728f5914a97e33160f899c39d4a6"
Oct 29 22:14:30.938985 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.938962    4550 scope.go:122] "RemoveContainer" containerID="33c3422c262f448be1cfdf14a0d935e658d0728f5914a97e33160f899c39d4a6"
Oct 29 22:14:30.939321 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:30.939301    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c3422c262f448be1cfdf14a0d935e658d0728f5914a97e33160f899c39d4a6\": container with ID starting with 33c3422c262f448be1cfdf14a0d935e658d0728f5914a97e33160f899c39d4a6 not found: ID does not exist" containerID="33c3422c262f448be1cfdf14a0d935e658d0728f5914a97e33160f899c39d4a6"
Oct 29 22:14:30.939396 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.939330    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c3422c262f448be1cfdf14a0d935e658d0728f5914a97e33160f899c39d4a6"} err="failed to get container status \"33c3422c262f448be1cfdf14a0d935e658d0728f5914a97e33160f899c39d4a6\": rpc error: code = NotFound desc = could not find container \"33c3422c262f448be1cfdf14a0d935e658d0728f5914a97e33160f899c39d4a6\": container with ID starting with 33c3422c262f448be1cfdf14a0d935e658d0728f5914a97e33160f899c39d4a6 not found: ID does not exist"
Oct 29 22:14:30.990856 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.990821    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"]
Oct 29 22:14:30.995074 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:30.995041    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-qm22h"]
Oct 29 22:14:31.029093 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:31.029060    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"]
Oct 29 22:14:31.033245 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:31.033214    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-xvds7"]
Oct 29 22:14:31.065738 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:31.065368    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"]
Oct 29 22:14:31.069650 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:31.069589    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-684wk"]
Oct 29 22:14:31.567600 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:31.567552    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:31.993613 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:31.993577    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2913dd75-7d51-4ee4-b705-f6a7396d75de" path="/var/lib/kubelet/pods/2913dd75-7d51-4ee4-b705-f6a7396d75de/volumes"
Oct 29 22:14:31.994062 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:31.994048    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be3e55fd-83b9-4c52-a2a9-28da7d04477e" path="/var/lib/kubelet/pods/be3e55fd-83b9-4c52-a2a9-28da7d04477e/volumes"
Oct 29 22:14:31.994506 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:31.994491    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" path="/var/lib/kubelet/pods/e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9/volumes"
Oct 29 22:14:32.932189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:32.931288    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"]
Oct 29 22:14:32.932671 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:32.932624    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96" podUID="3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d" containerName="maas-api" containerID="cri-o://d2b097890345faba1a69a3aedcdcf93c381f7102b836e5cc1454bf2b0cf2c1da" gracePeriod=30
Oct 29 22:14:33.073190 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.073127    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:33.422371 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.411437    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:33.539472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.539118    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 22:14:33.715739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.715648    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:33.743824 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.743792    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-openshift-service-ca\") pod \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") "
Oct 29 22:14:33.767393 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.767362    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-metrics-certs\") pod \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") "
Oct 29 22:14:33.767693 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.767675    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-maas-api-tls\") pod \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") "
Oct 29 22:14:33.767838 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.767824    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-kube-api-access-2shpm\" (UniqueName: \"kubernetes.io/projected/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-kube-api-access-2shpm\") pod \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\" (UID: \"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d\") "
Oct 29 22:14:33.768474 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.768424    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-openshift-service-ca" pod "3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d" (UID: "3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:33.791434 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.790469    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-maas-api-tls" pod "3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d" (UID: "3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:33.791434 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.790536    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-metrics-certs" pod "3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d" (UID: "3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:33.793240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.793213    4550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-openshift-service-ca\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:33.793463 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.793438    4550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-metrics-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:33.793581 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.793569    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-maas-api-tls\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:33.801725 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.800855    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-kube-api-access-2shpm" pod "3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d" (UID: "3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d"). InnerVolumeSpecName "kube-api-access-2shpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:33.836439 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.836299    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:33.894841 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:33.894806    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2shpm\" (UniqueName: \"kubernetes.io/projected/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d-kube-api-access-2shpm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:34.015421 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:34.015326    4550 generic.go:356] "Generic (PLEG): container finished" podID="3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d" containerID="d2b097890345faba1a69a3aedcdcf93c381f7102b836e5cc1454bf2b0cf2c1da" exitCode=0
Oct 29 22:14:34.016489 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:34.016470    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"
Oct 29 22:14:34.028375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:34.028335    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96" event={"ID":"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d","Type":"ContainerDied","Data":"d2b097890345faba1a69a3aedcdcf93c381f7102b836e5cc1454bf2b0cf2c1da"}
Oct 29 22:14:34.028668 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:34.028626    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96" event={"ID":"3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d","Type":"ContainerDied","Data":"28fe4867adfb455654109acdd5d0803a9a975aa3d668b4a996839369dd37ba66"}
Oct 29 22:14:34.028803 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:34.028788    4550 scope.go:122] "RemoveContainer" containerID="d2b097890345faba1a69a3aedcdcf93c381f7102b836e5cc1454bf2b0cf2c1da"
Oct 29 22:14:34.067427 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:34.067404    4550 scope.go:122] "RemoveContainer" containerID="d2b097890345faba1a69a3aedcdcf93c381f7102b836e5cc1454bf2b0cf2c1da"
Oct 29 22:14:34.068608 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:34.068136    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b097890345faba1a69a3aedcdcf93c381f7102b836e5cc1454bf2b0cf2c1da\": container with ID starting with d2b097890345faba1a69a3aedcdcf93c381f7102b836e5cc1454bf2b0cf2c1da not found: ID does not exist" containerID="d2b097890345faba1a69a3aedcdcf93c381f7102b836e5cc1454bf2b0cf2c1da"
Oct 29 22:14:34.068771 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:34.068745    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b097890345faba1a69a3aedcdcf93c381f7102b836e5cc1454bf2b0cf2c1da"} err="failed to get container status \"d2b097890345faba1a69a3aedcdcf93c381f7102b836e5cc1454bf2b0cf2c1da\": rpc error: code = NotFound desc = could not find container \"d2b097890345faba1a69a3aedcdcf93c381f7102b836e5cc1454bf2b0cf2c1da\": container with ID starting with d2b097890345faba1a69a3aedcdcf93c381f7102b836e5cc1454bf2b0cf2c1da not found: ID does not exist"
Oct 29 22:14:34.097422 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:34.097385    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"]
Oct 29 22:14:34.100692 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:34.100661    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-743a5e-75f8f6644f-ktr96"]
Oct 29 22:14:34.222972 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:34.222942    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:34.345431 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:34.345392    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:34.473103 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:34.473071    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:34.614845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:34.614753    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:35.019352 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:35.019270    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:35.990643 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:35.990601    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d" path="/var/lib/kubelet/pods/3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d/volumes"
Oct 29 22:14:36.413520 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:36.413489    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:37.023349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:37.023319    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:37.144913 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:37.144877    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:37.942166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:37.941638    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:38.104708 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:38.104679    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:38.284174 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:38.284086    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:38.427563 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:38.427527    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:38.457858 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:38.457824    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"]
Oct 29 22:14:38.458083 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:38.458055    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerName="istio-proxy" containerID="cri-o://7d7ca906a936c6886783427317e3064c6881c3da32b11a21f526989c1999b7b2" gracePeriod=30
Oct 29 22:14:38.464991 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:38.464964    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"]
Oct 29 22:14:38.465214 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:38.465192    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy" containerID="cri-o://e0e901984945f548fc6df40c334a1037b2c6d5d31c857ed14d0ba4d069da0c7a" gracePeriod=30
Oct 29 22:14:38.468257 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:38.467819    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"]
Oct 29 22:14:38.468257 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:38.468004    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerName="istio-proxy" containerID="cri-o://be88438cf85dfcb626c515343616bbecfba1c8709d0531fdd21a87275daa7d98" gracePeriod=30
Oct 29 22:14:38.614827 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:38.573315    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:40.677409 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:40.677369    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:41.162064 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:41.162031    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:41.277456 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:41.277427    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:41.513676 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:41.513575    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"]
Oct 29 22:14:41.513945 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:41.513916    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf" podUID="db332eb5-5ced-43e9-953b-5d94aa37d706" containerName="istio-proxy" containerID="cri-o://6ca09127de40590aa8ab387ad1b33780deb7369993fd708742ee536221a1281f" gracePeriod=30
Oct 29 22:14:41.530804 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:41.530770    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"]
Oct 29 22:14:41.532971 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:41.532112    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49" podUID="37c48198-27e2-4aa6-86e7-389bd62a2666" containerName="istio-proxy" containerID="cri-o://fbfa9b2f6c802c7485d2468e40c67f974deec08499ccd2cd5bfcab64d0f317f4" gracePeriod=30
Oct 29 22:14:41.541478 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:41.541337    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"]
Oct 29 22:14:41.541959 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:41.541931    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp" podUID="d6233b12-31ad-41d0-bb31-12b2f71161bb" containerName="istio-proxy" containerID="cri-o://18c2bba4296d509437428577b04f13ff465481fe334118306f474a9e8c5183b9" gracePeriod=30
Oct 29 22:14:41.548937 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:41.548916    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:14:41.550742 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:41.550720    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:42.077110 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:42.077075    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:42.280022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:42.279989    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:42.434118 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:42.434034    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"]
Oct 29 22:14:42.434588 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:42.434553    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p" podUID="b27b23e3-07bd-4531-9d64-73d1b66de485" containerName="istio-proxy" containerID="cri-o://026efd88a94b6a2b6b5cec91b23e0891a54175126ba444ffede798b542f2635b" gracePeriod=30
Oct 29 22:14:42.453535 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:42.453508    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:42.590165 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:42.584800    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:42.991814 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:42.991780    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:43.468184 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.468131    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:43.468570 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.468224    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf" podUID="db332eb5-5ced-43e9-953b-5d94aa37d706" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:43.765887 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.765578    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 22:14:43.780363 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.780324    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-workload-certs\") pod \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") "
Oct 29 22:14:43.780549 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.780394    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-workload-socket\") pod \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") "
Oct 29 22:14:43.780549 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.780466    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-kube-api-access-zp8ds\" (UniqueName: \"kubernetes.io/projected/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-kube-api-access-zp8ds\") pod \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") "
Oct 29 22:14:43.780549 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.780543    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-podinfo\") pod \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") "
Oct 29 22:14:43.780755 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.780588    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-data\") pod \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") "
Oct 29 22:14:43.780755 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.780688    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-token\" (UniqueName: \"kubernetes.io/projected/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-token\") pod \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") "
Oct 29 22:14:43.780869 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.780776    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istiod-ca-cert\") pod \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") "
Oct 29 22:14:43.780869 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.780829    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-envoy\") pod \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") "
Oct 29 22:14:43.780976 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.780923    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-credential-socket\") pod \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\" (UID: \"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1\") "
Oct 29 22:14:43.782633 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.782364    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-data" pod "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" (UID: "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.782950 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.782887    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-credential-socket" pod "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" (UID: "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.783053 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.782956    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-workload-certs" pod "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" (UID: "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.783124 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.783108    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istiod-ca-cert" pod "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" (UID: "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:43.783210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.783183    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-workload-socket" pod "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" (UID: "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.787272 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.787231    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-kube-api-access-zp8ds" pod "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" (UID: "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1"). InnerVolumeSpecName "kube-api-access-zp8ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:43.787999 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.787965    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-token" pod "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" (UID: "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:43.788615 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.788594    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-podinfo" pod "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" (UID: "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:43.789045 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.789022    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-envoy" pod "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" (UID: "9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.857255 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.857196    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 22:14:43.859968 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.859940    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 22:14:43.882473 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.882442    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-credential-socket\") pod \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") "
Oct 29 22:14:43.882668 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.882498    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istiod-ca-cert\") pod \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") "
Oct 29 22:14:43.882668 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.882537    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-data\") pod \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") "
Oct 29 22:14:43.882668 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.882563    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-token\" (UniqueName: \"kubernetes.io/projected/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-token\") pod \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") "
Oct 29 22:14:43.882668 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.882658    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-podinfo\") pod \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") "
Oct 29 22:14:43.882887 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.882696    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-workload-socket\") pod \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") "
Oct 29 22:14:43.882887 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.882725    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-credential-socket\") pod \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") "
Oct 29 22:14:43.882887 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.882725    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-credential-socket" pod "5058fbcb-91cf-44bd-9d4c-34f2acc83f23" (UID: "5058fbcb-91cf-44bd-9d4c-34f2acc83f23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.882887 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.882763    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-envoy\") pod \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") "
Oct 29 22:14:43.882887 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.882839    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-kube-api-access-49hzn\" (UniqueName: \"kubernetes.io/projected/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-kube-api-access-49hzn\") pod \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") "
Oct 29 22:14:43.882887 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.882874    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-workload-certs\") pod \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") "
Oct 29 22:14:43.883221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.882902    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-data\") pod \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") "
Oct 29 22:14:43.883221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.882950    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-envoy\") pod \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") "
Oct 29 22:14:43.883221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.883029    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-workload-socket\") pod \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") "
Oct 29 22:14:43.883221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.883055    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-podinfo\") pod \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") "
Oct 29 22:14:43.883221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.883088    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-kube-api-access-7xxq2\" (UniqueName: \"kubernetes.io/projected/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-kube-api-access-7xxq2\") pod \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") "
Oct 29 22:14:43.883221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.883120    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-workload-certs\") pod \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\" (UID: \"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c\") "
Oct 29 22:14:43.883221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.883174    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istiod-ca-cert\") pod \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") "
Oct 29 22:14:43.883221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.883210    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-token\" (UniqueName: \"kubernetes.io/projected/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-token\") pod \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\" (UID: \"5058fbcb-91cf-44bd-9d4c-34f2acc83f23\") "
Oct 29 22:14:43.883687 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.883389    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istiod-ca-cert" pod "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" (UID: "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:43.884010 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.883984    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-credential-socket" pod "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" (UID: "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.884236 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.884212    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-workload-socket" pod "5058fbcb-91cf-44bd-9d4c-34f2acc83f23" (UID: "5058fbcb-91cf-44bd-9d4c-34f2acc83f23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.884313 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.884248    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-data" pod "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" (UID: "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.884538    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.884566    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.884582    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.884598    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.884616    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp8ds\" (UniqueName: \"kubernetes.io/projected/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-kube-api-access-zp8ds\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.884631    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.884659    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.884674    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.884687    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.884703    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.884719    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.884731    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.885008    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-workload-certs" pod "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" (UID: "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.885217    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-data" pod "5058fbcb-91cf-44bd-9d4c-34f2acc83f23" (UID: "5058fbcb-91cf-44bd-9d4c-34f2acc83f23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.885364    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-podinfo" pod "5058fbcb-91cf-44bd-9d4c-34f2acc83f23" (UID: "5058fbcb-91cf-44bd-9d4c-34f2acc83f23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.885591    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-workload-certs" pod "5058fbcb-91cf-44bd-9d4c-34f2acc83f23" (UID: "5058fbcb-91cf-44bd-9d4c-34f2acc83f23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.885932    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istiod-ca-cert" pod "5058fbcb-91cf-44bd-9d4c-34f2acc83f23" (UID: "5058fbcb-91cf-44bd-9d4c-34f2acc83f23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:43.886302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.886268    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-workload-socket" pod "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" (UID: "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.887460 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.886747    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-kube-api-access-7xxq2" pod "5058fbcb-91cf-44bd-9d4c-34f2acc83f23" (UID: "5058fbcb-91cf-44bd-9d4c-34f2acc83f23"). InnerVolumeSpecName "kube-api-access-7xxq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:43.887460 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.886990    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-kube-api-access-49hzn" pod "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" (UID: "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c"). InnerVolumeSpecName "kube-api-access-49hzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:43.887613 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.887593    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-envoy" pod "5058fbcb-91cf-44bd-9d4c-34f2acc83f23" (UID: "5058fbcb-91cf-44bd-9d4c-34f2acc83f23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.888002 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.887946    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-podinfo" pod "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" (UID: "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:43.888274 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.888247    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-token" pod "5058fbcb-91cf-44bd-9d4c-34f2acc83f23" (UID: "5058fbcb-91cf-44bd-9d4c-34f2acc83f23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:43.889871 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.889834    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-token" pod "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" (UID: "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:43.894054 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.894022    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-envoy" pod "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" (UID: "4f0953b3-a4b1-425f-bf8c-1b0dc847b38c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.936284 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.936254    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:43.936425 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.936301    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp" podUID="d6233b12-31ad-41d0-bb31-12b2f71161bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:43.986036 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.985956    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.986036 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.985981    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.986036 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.985991    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.986036 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.986000    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.986036 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.986008    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.986036 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.986019    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49hzn\" (UniqueName: \"kubernetes.io/projected/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-kube-api-access-49hzn\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.986036 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.986033    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.986036 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.986041    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.986604 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.986049    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.986604 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.986056    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.986604 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.986065    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.986604 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.986075    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xxq2\" (UniqueName: \"kubernetes.io/projected/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-kube-api-access-7xxq2\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.986604 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.986084    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.986604 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.986097    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.986604 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:43.986107    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5058fbcb-91cf-44bd-9d4c-34f2acc83f23-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.082844 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.082805    4550 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:44.083066 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.082876    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49" podUID="37c48198-27e2-4aa6-86e7-389bd62a2666" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:44.095609 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.095575    4550 generic.go:356] "Generic (PLEG): container finished" podID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerID="7d7ca906a936c6886783427317e3064c6881c3da32b11a21f526989c1999b7b2" exitCode=0
Oct 29 22:14:44.095796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.095642    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"
Oct 29 22:14:44.095796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.095681    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" event={"ID":"5058fbcb-91cf-44bd-9d4c-34f2acc83f23","Type":"ContainerDied","Data":"7d7ca906a936c6886783427317e3064c6881c3da32b11a21f526989c1999b7b2"}
Oct 29 22:14:44.095796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.095760    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl" event={"ID":"5058fbcb-91cf-44bd-9d4c-34f2acc83f23","Type":"ContainerDied","Data":"ba7935c5473e6fd9a179568fb2169eb14d15544c5efb16ff8071248001c858fa"}
Oct 29 22:14:44.095796 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.095796    4550 scope.go:122] "RemoveContainer" containerID="7d7ca906a936c6886783427317e3064c6881c3da32b11a21f526989c1999b7b2"
Oct 29 22:14:44.097939 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.097916    4550 generic.go:356] "Generic (PLEG): container finished" podID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerID="be88438cf85dfcb626c515343616bbecfba1c8709d0531fdd21a87275daa7d98" exitCode=0
Oct 29 22:14:44.098466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.098443    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" event={"ID":"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1","Type":"ContainerDied","Data":"be88438cf85dfcb626c515343616bbecfba1c8709d0531fdd21a87275daa7d98"}
Oct 29 22:14:44.098620 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.098601    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf" event={"ID":"9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1","Type":"ContainerDied","Data":"57c8355070ec842d63d3376bf1302aa35e781c0dd170e69592e62c4ec86bdc73"}
Oct 29 22:14:44.100230 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.100187    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"
Oct 29 22:14:44.101751 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.101723    4550 generic.go:356] "Generic (PLEG): container finished" podID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerID="e0e901984945f548fc6df40c334a1037b2c6d5d31c857ed14d0ba4d069da0c7a" exitCode=0
Oct 29 22:14:44.101871 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.101837    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" event={"ID":"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c","Type":"ContainerDied","Data":"e0e901984945f548fc6df40c334a1037b2c6d5d31c857ed14d0ba4d069da0c7a"}
Oct 29 22:14:44.101871 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.101854    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"
Oct 29 22:14:44.101990 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.101875    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9" event={"ID":"4f0953b3-a4b1-425f-bf8c-1b0dc847b38c","Type":"ContainerDied","Data":"1815cf6c397098e29cd86c9c2f5378971a5c64527b581b31253049c3c7391a70"}
Oct 29 22:14:44.110209 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.110190    4550 scope.go:122] "RemoveContainer" containerID="7d7ca906a936c6886783427317e3064c6881c3da32b11a21f526989c1999b7b2"
Oct 29 22:14:44.112108 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:44.112080    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7ca906a936c6886783427317e3064c6881c3da32b11a21f526989c1999b7b2\": container with ID starting with 7d7ca906a936c6886783427317e3064c6881c3da32b11a21f526989c1999b7b2 not found: ID does not exist" containerID="7d7ca906a936c6886783427317e3064c6881c3da32b11a21f526989c1999b7b2"
Oct 29 22:14:44.112281 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.112252    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7ca906a936c6886783427317e3064c6881c3da32b11a21f526989c1999b7b2"} err="failed to get container status \"7d7ca906a936c6886783427317e3064c6881c3da32b11a21f526989c1999b7b2\": rpc error: code = NotFound desc = could not find container \"7d7ca906a936c6886783427317e3064c6881c3da32b11a21f526989c1999b7b2\": container with ID starting with 7d7ca906a936c6886783427317e3064c6881c3da32b11a21f526989c1999b7b2 not found: ID does not exist"
Oct 29 22:14:44.112393 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.112380    4550 scope.go:122] "RemoveContainer" containerID="be88438cf85dfcb626c515343616bbecfba1c8709d0531fdd21a87275daa7d98"
Oct 29 22:14:44.123422 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.123400    4550 scope.go:122] "RemoveContainer" containerID="be88438cf85dfcb626c515343616bbecfba1c8709d0531fdd21a87275daa7d98"
Oct 29 22:14:44.123792 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:44.123764    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be88438cf85dfcb626c515343616bbecfba1c8709d0531fdd21a87275daa7d98\": container with ID starting with be88438cf85dfcb626c515343616bbecfba1c8709d0531fdd21a87275daa7d98 not found: ID does not exist" containerID="be88438cf85dfcb626c515343616bbecfba1c8709d0531fdd21a87275daa7d98"
Oct 29 22:14:44.123918 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.123802    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be88438cf85dfcb626c515343616bbecfba1c8709d0531fdd21a87275daa7d98"} err="failed to get container status \"be88438cf85dfcb626c515343616bbecfba1c8709d0531fdd21a87275daa7d98\": rpc error: code = NotFound desc = could not find container \"be88438cf85dfcb626c515343616bbecfba1c8709d0531fdd21a87275daa7d98\": container with ID starting with be88438cf85dfcb626c515343616bbecfba1c8709d0531fdd21a87275daa7d98 not found: ID does not exist"
Oct 29 22:14:44.123918 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.123825    4550 scope.go:122] "RemoveContainer" containerID="e0e901984945f548fc6df40c334a1037b2c6d5d31c857ed14d0ba4d069da0c7a"
Oct 29 22:14:44.135295 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.135217    4550 scope.go:122] "RemoveContainer" containerID="e0e901984945f548fc6df40c334a1037b2c6d5d31c857ed14d0ba4d069da0c7a"
Oct 29 22:14:44.135556 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:44.135527    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e901984945f548fc6df40c334a1037b2c6d5d31c857ed14d0ba4d069da0c7a\": container with ID starting with e0e901984945f548fc6df40c334a1037b2c6d5d31c857ed14d0ba4d069da0c7a not found: ID does not exist" containerID="e0e901984945f548fc6df40c334a1037b2c6d5d31c857ed14d0ba4d069da0c7a"
Oct 29 22:14:44.135625 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.135566    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e901984945f548fc6df40c334a1037b2c6d5d31c857ed14d0ba4d069da0c7a"} err="failed to get container status \"e0e901984945f548fc6df40c334a1037b2c6d5d31c857ed14d0ba4d069da0c7a\": rpc error: code = NotFound desc = could not find container \"e0e901984945f548fc6df40c334a1037b2c6d5d31c857ed14d0ba4d069da0c7a\": container with ID starting with e0e901984945f548fc6df40c334a1037b2c6d5d31c857ed14d0ba4d069da0c7a not found: ID does not exist"
Oct 29 22:14:44.151081 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.151059    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"]
Oct 29 22:14:44.156712 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.156690    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-jrvvl"]
Oct 29 22:14:44.192044 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.192015    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"]
Oct 29 22:14:44.197681 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.197648    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-n6wbf"]
Oct 29 22:14:44.231251 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.231225    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"]
Oct 29 22:14:44.236851 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.236784    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wk4p9"]
Oct 29 22:14:44.513539 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:44.513502    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:45.449886 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:45.449828    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:45.563587 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:45.563553    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:45.988488 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:45.988457    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" path="/var/lib/kubelet/pods/4f0953b3-a4b1-425f-bf8c-1b0dc847b38c/volumes"
Oct 29 22:14:45.988965 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:45.988952    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" path="/var/lib/kubelet/pods/5058fbcb-91cf-44bd-9d4c-34f2acc83f23/volumes"
Oct 29 22:14:45.989418 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:45.989405    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" path="/var/lib/kubelet/pods/9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1/volumes"
Oct 29 22:14:46.339395 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.339363    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:46.857347 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.841460    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 22:14:46.888375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.888337    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:14:46.912467 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.912444    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:14:46.923339 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.923314    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-envoy\") pod \"db332eb5-5ced-43e9-953b-5d94aa37d706\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") "
Oct 29 22:14:46.923459 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.923403    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-envoy\") pod \"37c48198-27e2-4aa6-86e7-389bd62a2666\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") "
Oct 29 22:14:46.923559 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.923470    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-workload-socket\") pod \"37c48198-27e2-4aa6-86e7-389bd62a2666\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") "
Oct 29 22:14:46.923764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.923737    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-workload-socket" pod "37c48198-27e2-4aa6-86e7-389bd62a2666" (UID: "37c48198-27e2-4aa6-86e7-389bd62a2666"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.923764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.923753    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db332eb5-5ced-43e9-953b-5d94aa37d706-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db332eb5-5ced-43e9-953b-5d94aa37d706-istiod-ca-cert\") pod \"db332eb5-5ced-43e9-953b-5d94aa37d706\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") "
Oct 29 22:14:46.923952 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.923800    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-token\" (UniqueName: \"kubernetes.io/projected/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-token\") pod \"db332eb5-5ced-43e9-953b-5d94aa37d706\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") "
Oct 29 22:14:46.923952 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.923913    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-workload-certs\") pod \"37c48198-27e2-4aa6-86e7-389bd62a2666\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") "
Oct 29 22:14:46.924072 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.923984    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37c48198-27e2-4aa6-86e7-389bd62a2666-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c48198-27e2-4aa6-86e7-389bd62a2666-istiod-ca-cert\") pod \"37c48198-27e2-4aa6-86e7-389bd62a2666\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") "
Oct 29 22:14:46.924072 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924042    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-credential-socket\") pod \"db332eb5-5ced-43e9-953b-5d94aa37d706\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") "
Oct 29 22:14:46.924233 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924084    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-data\") pod \"37c48198-27e2-4aa6-86e7-389bd62a2666\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") "
Oct 29 22:14:46.924233 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924115    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-token\" (UniqueName: \"kubernetes.io/projected/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-token\") pod \"37c48198-27e2-4aa6-86e7-389bd62a2666\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") "
Oct 29 22:14:46.924233 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924125    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db332eb5-5ced-43e9-953b-5d94aa37d706-istiod-ca-cert" pod "db332eb5-5ced-43e9-953b-5d94aa37d706" (UID: "db332eb5-5ced-43e9-953b-5d94aa37d706"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:46.924233 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924193    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-podinfo\") pod \"db332eb5-5ced-43e9-953b-5d94aa37d706\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") "
Oct 29 22:14:46.924438 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924237    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-data\") pod \"db332eb5-5ced-43e9-953b-5d94aa37d706\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") "
Oct 29 22:14:46.924438 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924305    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db332eb5-5ced-43e9-953b-5d94aa37d706-kube-api-access-hnrs2\" (UniqueName: \"kubernetes.io/projected/db332eb5-5ced-43e9-953b-5d94aa37d706-kube-api-access-hnrs2\") pod \"db332eb5-5ced-43e9-953b-5d94aa37d706\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") "
Oct 29 22:14:46.924438 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924358    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-podinfo\") pod \"37c48198-27e2-4aa6-86e7-389bd62a2666\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") "
Oct 29 22:14:46.924438 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924399    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-workload-socket\") pod \"db332eb5-5ced-43e9-953b-5d94aa37d706\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") "
Oct 29 22:14:46.924664 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924451    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-workload-certs\") pod \"db332eb5-5ced-43e9-953b-5d94aa37d706\" (UID: \"db332eb5-5ced-43e9-953b-5d94aa37d706\") "
Oct 29 22:14:46.924664 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924497    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c48198-27e2-4aa6-86e7-389bd62a2666-kube-api-access-mp922\" (UniqueName: \"kubernetes.io/projected/37c48198-27e2-4aa6-86e7-389bd62a2666-kube-api-access-mp922\") pod \"37c48198-27e2-4aa6-86e7-389bd62a2666\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") "
Oct 29 22:14:46.924664 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924516    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37c48198-27e2-4aa6-86e7-389bd62a2666-istiod-ca-cert" pod "37c48198-27e2-4aa6-86e7-389bd62a2666" (UID: "37c48198-27e2-4aa6-86e7-389bd62a2666"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:46.924664 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924525    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-credential-socket\") pod \"37c48198-27e2-4aa6-86e7-389bd62a2666\" (UID: \"37c48198-27e2-4aa6-86e7-389bd62a2666\") "
Oct 29 22:14:46.924664 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924612    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-workload-certs" pod "37c48198-27e2-4aa6-86e7-389bd62a2666" (UID: "37c48198-27e2-4aa6-86e7-389bd62a2666"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.924919 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924769    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-credential-socket" pod "db332eb5-5ced-43e9-953b-5d94aa37d706" (UID: "db332eb5-5ced-43e9-953b-5d94aa37d706"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.924919 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.924821    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-data" pod "37c48198-27e2-4aa6-86e7-389bd62a2666" (UID: "37c48198-27e2-4aa6-86e7-389bd62a2666"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.925231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.925210    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.925231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.925225    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-workload-socket" pod "db332eb5-5ced-43e9-953b-5d94aa37d706" (UID: "db332eb5-5ced-43e9-953b-5d94aa37d706"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.925370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.925237    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db332eb5-5ced-43e9-953b-5d94aa37d706-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.925370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.925251    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.925370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.925265    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c48198-27e2-4aa6-86e7-389bd62a2666-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.925370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.925278    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.925370 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.925293    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.926126 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.925898    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-data" pod "db332eb5-5ced-43e9-953b-5d94aa37d706" (UID: "db332eb5-5ced-43e9-953b-5d94aa37d706"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.926126 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.925953    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-workload-certs" pod "db332eb5-5ced-43e9-953b-5d94aa37d706" (UID: "db332eb5-5ced-43e9-953b-5d94aa37d706"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.926126 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.926091    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-credential-socket" pod "37c48198-27e2-4aa6-86e7-389bd62a2666" (UID: "37c48198-27e2-4aa6-86e7-389bd62a2666"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.932657 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.932625    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-podinfo" pod "37c48198-27e2-4aa6-86e7-389bd62a2666" (UID: "37c48198-27e2-4aa6-86e7-389bd62a2666"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:46.932657 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.932655    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c48198-27e2-4aa6-86e7-389bd62a2666-kube-api-access-mp922" pod "37c48198-27e2-4aa6-86e7-389bd62a2666" (UID: "37c48198-27e2-4aa6-86e7-389bd62a2666"). InnerVolumeSpecName "kube-api-access-mp922". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.932815 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.932635    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db332eb5-5ced-43e9-953b-5d94aa37d706-kube-api-access-hnrs2" pod "db332eb5-5ced-43e9-953b-5d94aa37d706" (UID: "db332eb5-5ced-43e9-953b-5d94aa37d706"). InnerVolumeSpecName "kube-api-access-hnrs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.932815 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.932768    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-envoy" pod "37c48198-27e2-4aa6-86e7-389bd62a2666" (UID: "37c48198-27e2-4aa6-86e7-389bd62a2666"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.932815 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.932767    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-token" pod "37c48198-27e2-4aa6-86e7-389bd62a2666" (UID: "37c48198-27e2-4aa6-86e7-389bd62a2666"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.932815 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.932799    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-token" pod "db332eb5-5ced-43e9-953b-5d94aa37d706" (UID: "db332eb5-5ced-43e9-953b-5d94aa37d706"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.932815 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.932810    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-envoy" pod "db332eb5-5ced-43e9-953b-5d94aa37d706" (UID: "db332eb5-5ced-43e9-953b-5d94aa37d706"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.934009 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:46.933985    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-podinfo" pod "db332eb5-5ced-43e9-953b-5d94aa37d706" (UID: "db332eb5-5ced-43e9-953b-5d94aa37d706"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:47.026599 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.026564    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-workload-socket\") pod \"d6233b12-31ad-41d0-bb31-12b2f71161bb\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") "
Oct 29 22:14:47.026912 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.026863    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-workload-socket" pod "d6233b12-31ad-41d0-bb31-12b2f71161bb" (UID: "d6233b12-31ad-41d0-bb31-12b2f71161bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.026912 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.026889    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-token\" (UniqueName: \"kubernetes.io/projected/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-token\") pod \"d6233b12-31ad-41d0-bb31-12b2f71161bb\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") "
Oct 29 22:14:47.027131 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.026943    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6233b12-31ad-41d0-bb31-12b2f71161bb-kube-api-access-6sp9s\" (UniqueName: \"kubernetes.io/projected/d6233b12-31ad-41d0-bb31-12b2f71161bb-kube-api-access-6sp9s\") pod \"d6233b12-31ad-41d0-bb31-12b2f71161bb\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") "
Oct 29 22:14:47.027131 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.026986    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-podinfo\") pod \"d6233b12-31ad-41d0-bb31-12b2f71161bb\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") "
Oct 29 22:14:47.027131 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.027078    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-credential-socket\") pod \"d6233b12-31ad-41d0-bb31-12b2f71161bb\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") "
Oct 29 22:14:47.027131 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.027108    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-envoy\") pod \"d6233b12-31ad-41d0-bb31-12b2f71161bb\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") "
Oct 29 22:14:47.027551 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.027180    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6233b12-31ad-41d0-bb31-12b2f71161bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6233b12-31ad-41d0-bb31-12b2f71161bb-istiod-ca-cert\") pod \"d6233b12-31ad-41d0-bb31-12b2f71161bb\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") "
Oct 29 22:14:47.027551 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.027219    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-workload-certs\") pod \"d6233b12-31ad-41d0-bb31-12b2f71161bb\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") "
Oct 29 22:14:47.027551 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.027314    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-data\") pod \"d6233b12-31ad-41d0-bb31-12b2f71161bb\" (UID: \"d6233b12-31ad-41d0-bb31-12b2f71161bb\") "
Oct 29 22:14:47.028015 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.027663    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-data" pod "d6233b12-31ad-41d0-bb31-12b2f71161bb" (UID: "d6233b12-31ad-41d0-bb31-12b2f71161bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028393    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-credential-socket" pod "d6233b12-31ad-41d0-bb31-12b2f71161bb" (UID: "d6233b12-31ad-41d0-bb31-12b2f71161bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028678    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028704    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028719    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028732    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028746    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028759    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028771    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028783    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028796    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028808    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnrs2\" (UniqueName: \"kubernetes.io/projected/db332eb5-5ced-43e9-953b-5d94aa37d706-kube-api-access-hnrs2\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028824    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c48198-27e2-4aa6-86e7-389bd62a2666-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028837    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028849    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db332eb5-5ced-43e9-953b-5d94aa37d706-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028863    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp922\" (UniqueName: \"kubernetes.io/projected/37c48198-27e2-4aa6-86e7-389bd62a2666-kube-api-access-mp922\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.028896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.028876    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c48198-27e2-4aa6-86e7-389bd62a2666-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.031413 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.031257    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-workload-certs" pod "d6233b12-31ad-41d0-bb31-12b2f71161bb" (UID: "d6233b12-31ad-41d0-bb31-12b2f71161bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.031413 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.031352    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6233b12-31ad-41d0-bb31-12b2f71161bb-kube-api-access-6sp9s" pod "d6233b12-31ad-41d0-bb31-12b2f71161bb" (UID: "d6233b12-31ad-41d0-bb31-12b2f71161bb"). InnerVolumeSpecName "kube-api-access-6sp9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:47.031413 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.031388    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-token" pod "d6233b12-31ad-41d0-bb31-12b2f71161bb" (UID: "d6233b12-31ad-41d0-bb31-12b2f71161bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:47.031413 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.031392    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6233b12-31ad-41d0-bb31-12b2f71161bb-istiod-ca-cert" pod "d6233b12-31ad-41d0-bb31-12b2f71161bb" (UID: "d6233b12-31ad-41d0-bb31-12b2f71161bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:47.033237 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.033207    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-envoy" pod "d6233b12-31ad-41d0-bb31-12b2f71161bb" (UID: "d6233b12-31ad-41d0-bb31-12b2f71161bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.033414 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.033394    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-podinfo" pod "d6233b12-31ad-41d0-bb31-12b2f71161bb" (UID: "d6233b12-31ad-41d0-bb31-12b2f71161bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:47.123820 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.123731    4550 generic.go:356] "Generic (PLEG): container finished" podID="d6233b12-31ad-41d0-bb31-12b2f71161bb" containerID="18c2bba4296d509437428577b04f13ff465481fe334118306f474a9e8c5183b9" exitCode=0
Oct 29 22:14:47.123820 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.123778    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"
Oct 29 22:14:47.124033 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.123815    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp" event={"ID":"d6233b12-31ad-41d0-bb31-12b2f71161bb","Type":"ContainerDied","Data":"18c2bba4296d509437428577b04f13ff465481fe334118306f474a9e8c5183b9"}
Oct 29 22:14:47.124033 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.123869    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp" event={"ID":"d6233b12-31ad-41d0-bb31-12b2f71161bb","Type":"ContainerDied","Data":"151a2a779943e53b0b229fcd3574ba07f9014259a2144091b0a5de30d553a3da"}
Oct 29 22:14:47.124033 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.123892    4550 scope.go:122] "RemoveContainer" containerID="18c2bba4296d509437428577b04f13ff465481fe334118306f474a9e8c5183b9"
Oct 29 22:14:47.125320 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.125300    4550 generic.go:356] "Generic (PLEG): container finished" podID="db332eb5-5ced-43e9-953b-5d94aa37d706" containerID="6ca09127de40590aa8ab387ad1b33780deb7369993fd708742ee536221a1281f" exitCode=0
Oct 29 22:14:47.125439 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.125356    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"
Oct 29 22:14:47.125439 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.125376    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf" event={"ID":"db332eb5-5ced-43e9-953b-5d94aa37d706","Type":"ContainerDied","Data":"6ca09127de40590aa8ab387ad1b33780deb7369993fd708742ee536221a1281f"}
Oct 29 22:14:47.125439 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.125424    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf" event={"ID":"db332eb5-5ced-43e9-953b-5d94aa37d706","Type":"ContainerDied","Data":"cdeff4dd051f02ec6b052a4092c8b18c8c25e6b83898594b66595df20f67d820"}
Oct 29 22:14:47.126923 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.126893    4550 generic.go:356] "Generic (PLEG): container finished" podID="37c48198-27e2-4aa6-86e7-389bd62a2666" containerID="fbfa9b2f6c802c7485d2468e40c67f974deec08499ccd2cd5bfcab64d0f317f4" exitCode=0
Oct 29 22:14:47.127047 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.126968    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49" event={"ID":"37c48198-27e2-4aa6-86e7-389bd62a2666","Type":"ContainerDied","Data":"fbfa9b2f6c802c7485d2468e40c67f974deec08499ccd2cd5bfcab64d0f317f4"}
Oct 29 22:14:47.127047 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.127009    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49" event={"ID":"37c48198-27e2-4aa6-86e7-389bd62a2666","Type":"ContainerDied","Data":"365110503a5d2658688423db366116272e5b51020be6695734bd88611cae2fee"}
Oct 29 22:14:47.127173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.127073    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"
Oct 29 22:14:47.131012 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.130989    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.131114 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.131017    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sp9s\" (UniqueName: \"kubernetes.io/projected/d6233b12-31ad-41d0-bb31-12b2f71161bb-kube-api-access-6sp9s\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.131114 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.131032    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.131114 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.131045    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.131114 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.131059    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6233b12-31ad-41d0-bb31-12b2f71161bb-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.131114 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.131071    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6233b12-31ad-41d0-bb31-12b2f71161bb-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.135472 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.135448    4550 scope.go:122] "RemoveContainer" containerID="18c2bba4296d509437428577b04f13ff465481fe334118306f474a9e8c5183b9"
Oct 29 22:14:47.135769 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:47.135746    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c2bba4296d509437428577b04f13ff465481fe334118306f474a9e8c5183b9\": container with ID starting with 18c2bba4296d509437428577b04f13ff465481fe334118306f474a9e8c5183b9 not found: ID does not exist" containerID="18c2bba4296d509437428577b04f13ff465481fe334118306f474a9e8c5183b9"
Oct 29 22:14:47.135839 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.135780    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c2bba4296d509437428577b04f13ff465481fe334118306f474a9e8c5183b9"} err="failed to get container status \"18c2bba4296d509437428577b04f13ff465481fe334118306f474a9e8c5183b9\": rpc error: code = NotFound desc = could not find container \"18c2bba4296d509437428577b04f13ff465481fe334118306f474a9e8c5183b9\": container with ID starting with 18c2bba4296d509437428577b04f13ff465481fe334118306f474a9e8c5183b9 not found: ID does not exist"
Oct 29 22:14:47.135839 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.135802    4550 scope.go:122] "RemoveContainer" containerID="6ca09127de40590aa8ab387ad1b33780deb7369993fd708742ee536221a1281f"
Oct 29 22:14:47.145846 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.145827    4550 scope.go:122] "RemoveContainer" containerID="6ca09127de40590aa8ab387ad1b33780deb7369993fd708742ee536221a1281f"
Oct 29 22:14:47.146117 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:47.146098    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca09127de40590aa8ab387ad1b33780deb7369993fd708742ee536221a1281f\": container with ID starting with 6ca09127de40590aa8ab387ad1b33780deb7369993fd708742ee536221a1281f not found: ID does not exist" containerID="6ca09127de40590aa8ab387ad1b33780deb7369993fd708742ee536221a1281f"
Oct 29 22:14:47.146238 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.146125    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca09127de40590aa8ab387ad1b33780deb7369993fd708742ee536221a1281f"} err="failed to get container status \"6ca09127de40590aa8ab387ad1b33780deb7369993fd708742ee536221a1281f\": rpc error: code = NotFound desc = could not find container \"6ca09127de40590aa8ab387ad1b33780deb7369993fd708742ee536221a1281f\": container with ID starting with 6ca09127de40590aa8ab387ad1b33780deb7369993fd708742ee536221a1281f not found: ID does not exist"
Oct 29 22:14:47.146238 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.146178    4550 scope.go:122] "RemoveContainer" containerID="fbfa9b2f6c802c7485d2468e40c67f974deec08499ccd2cd5bfcab64d0f317f4"
Oct 29 22:14:47.157803 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.157786    4550 scope.go:122] "RemoveContainer" containerID="fbfa9b2f6c802c7485d2468e40c67f974deec08499ccd2cd5bfcab64d0f317f4"
Oct 29 22:14:47.158070 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:47.158051    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbfa9b2f6c802c7485d2468e40c67f974deec08499ccd2cd5bfcab64d0f317f4\": container with ID starting with fbfa9b2f6c802c7485d2468e40c67f974deec08499ccd2cd5bfcab64d0f317f4 not found: ID does not exist" containerID="fbfa9b2f6c802c7485d2468e40c67f974deec08499ccd2cd5bfcab64d0f317f4"
Oct 29 22:14:47.158129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.158079    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbfa9b2f6c802c7485d2468e40c67f974deec08499ccd2cd5bfcab64d0f317f4"} err="failed to get container status \"fbfa9b2f6c802c7485d2468e40c67f974deec08499ccd2cd5bfcab64d0f317f4\": rpc error: code = NotFound desc = could not find container \"fbfa9b2f6c802c7485d2468e40c67f974deec08499ccd2cd5bfcab64d0f317f4\": container with ID starting with fbfa9b2f6c802c7485d2468e40c67f974deec08499ccd2cd5bfcab64d0f317f4 not found: ID does not exist"
Oct 29 22:14:47.210495 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.210466    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"]
Oct 29 22:14:47.215785 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.215762    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-q56sp"]
Oct 29 22:14:47.253595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.253572    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"]
Oct 29 22:14:47.257426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.257396    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-bkxxf"]
Oct 29 22:14:47.294904 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.294874    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"]
Oct 29 22:14:47.297043 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.297018    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-cvm49"]
Oct 29 22:14:47.712608 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.712548    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:14:47.838661 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.838633    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-podinfo\") pod \"b27b23e3-07bd-4531-9d64-73d1b66de485\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") "
Oct 29 22:14:47.838951 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.838686    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b27b23e3-07bd-4531-9d64-73d1b66de485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b27b23e3-07bd-4531-9d64-73d1b66de485-istiod-ca-cert\") pod \"b27b23e3-07bd-4531-9d64-73d1b66de485\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") "
Oct 29 22:14:47.838951 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.838729    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-data\") pod \"b27b23e3-07bd-4531-9d64-73d1b66de485\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") "
Oct 29 22:14:47.838951 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.838759    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-workload-certs\") pod \"b27b23e3-07bd-4531-9d64-73d1b66de485\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") "
Oct 29 22:14:47.838951 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.838839    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-workload-socket\") pod \"b27b23e3-07bd-4531-9d64-73d1b66de485\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") "
Oct 29 22:14:47.838951 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.838881    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b27b23e3-07bd-4531-9d64-73d1b66de485-kube-api-access-hsm75\" (UniqueName: \"kubernetes.io/projected/b27b23e3-07bd-4531-9d64-73d1b66de485-kube-api-access-hsm75\") pod \"b27b23e3-07bd-4531-9d64-73d1b66de485\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") "
Oct 29 22:14:47.838951 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.838939    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-envoy\") pod \"b27b23e3-07bd-4531-9d64-73d1b66de485\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") "
Oct 29 22:14:47.839325 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.839035    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-credential-socket\") pod \"b27b23e3-07bd-4531-9d64-73d1b66de485\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") "
Oct 29 22:14:47.839325 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.839080    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-token\" (UniqueName: \"kubernetes.io/projected/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-token\") pod \"b27b23e3-07bd-4531-9d64-73d1b66de485\" (UID: \"b27b23e3-07bd-4531-9d64-73d1b66de485\") "
Oct 29 22:14:47.839325 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.839031    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-data" pod "b27b23e3-07bd-4531-9d64-73d1b66de485" (UID: "b27b23e3-07bd-4531-9d64-73d1b66de485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.839325 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.839122    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-workload-socket" pod "b27b23e3-07bd-4531-9d64-73d1b66de485" (UID: "b27b23e3-07bd-4531-9d64-73d1b66de485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.839325 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.839247    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-workload-certs" pod "b27b23e3-07bd-4531-9d64-73d1b66de485" (UID: "b27b23e3-07bd-4531-9d64-73d1b66de485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.839597 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.839349    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-credential-socket" pod "b27b23e3-07bd-4531-9d64-73d1b66de485" (UID: "b27b23e3-07bd-4531-9d64-73d1b66de485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.839597 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.839477    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27b23e3-07bd-4531-9d64-73d1b66de485-istiod-ca-cert" pod "b27b23e3-07bd-4531-9d64-73d1b66de485" (UID: "b27b23e3-07bd-4531-9d64-73d1b66de485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:47.839715 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.839641    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.839715 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.839657    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.839715 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.839671    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.839715 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.839690    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.839715 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.839709    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b27b23e3-07bd-4531-9d64-73d1b66de485-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.841416 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.841389    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-podinfo" pod "b27b23e3-07bd-4531-9d64-73d1b66de485" (UID: "b27b23e3-07bd-4531-9d64-73d1b66de485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:47.841544 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.841414    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-token" pod "b27b23e3-07bd-4531-9d64-73d1b66de485" (UID: "b27b23e3-07bd-4531-9d64-73d1b66de485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:47.841631 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.841615    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27b23e3-07bd-4531-9d64-73d1b66de485-kube-api-access-hsm75" pod "b27b23e3-07bd-4531-9d64-73d1b66de485" (UID: "b27b23e3-07bd-4531-9d64-73d1b66de485"). InnerVolumeSpecName "kube-api-access-hsm75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:47.841818 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.841798    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-envoy" pod "b27b23e3-07bd-4531-9d64-73d1b66de485" (UID: "b27b23e3-07bd-4531-9d64-73d1b66de485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.913020 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.912986    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"]
Oct 29 22:14:47.923356 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.923331    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"]
Oct 29 22:14:47.941348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.941314    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.941348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.941343    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.941348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.941356    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b27b23e3-07bd-4531-9d64-73d1b66de485-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.941614 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.941369    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsm75\" (UniqueName: \"kubernetes.io/projected/b27b23e3-07bd-4531-9d64-73d1b66de485-kube-api-access-hsm75\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.991742 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.991670    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c48198-27e2-4aa6-86e7-389bd62a2666" path="/var/lib/kubelet/pods/37c48198-27e2-4aa6-86e7-389bd62a2666/volumes"
Oct 29 22:14:47.992214 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.992195    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6233b12-31ad-41d0-bb31-12b2f71161bb" path="/var/lib/kubelet/pods/d6233b12-31ad-41d0-bb31-12b2f71161bb/volumes"
Oct 29 22:14:47.992651 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:47.992631    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db332eb5-5ced-43e9-953b-5d94aa37d706" path="/var/lib/kubelet/pods/db332eb5-5ced-43e9-953b-5d94aa37d706/volumes"
Oct 29 22:14:48.010308 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:48.010286    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:48.138344 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:48.138304    4550 generic.go:356] "Generic (PLEG): container finished" podID="b27b23e3-07bd-4531-9d64-73d1b66de485" containerID="026efd88a94b6a2b6b5cec91b23e0891a54175126ba444ffede798b542f2635b" exitCode=0
Oct 29 22:14:48.138572 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:48.138355    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p" event={"ID":"b27b23e3-07bd-4531-9d64-73d1b66de485","Type":"ContainerDied","Data":"026efd88a94b6a2b6b5cec91b23e0891a54175126ba444ffede798b542f2635b"}
Oct 29 22:14:48.138572 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:48.138384    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"
Oct 29 22:14:48.138572 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:48.138405    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p" event={"ID":"b27b23e3-07bd-4531-9d64-73d1b66de485","Type":"ContainerDied","Data":"59e3f45e3d8083afa4aad7fcee60e3d447a4370dcb7d8402d66bc1239e787a89"}
Oct 29 22:14:48.138572 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:48.138431    4550 scope.go:122] "RemoveContainer" containerID="026efd88a94b6a2b6b5cec91b23e0891a54175126ba444ffede798b542f2635b"
Oct 29 22:14:48.147964 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:48.147944    4550 scope.go:122] "RemoveContainer" containerID="3b1a1e2579e8bf8fe938ff45559dae6bb7a27cd5f164817a07c4da911fa170cf"
Oct 29 22:14:48.158181 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:48.158157    4550 scope.go:122] "RemoveContainer" containerID="026efd88a94b6a2b6b5cec91b23e0891a54175126ba444ffede798b542f2635b"
Oct 29 22:14:48.158681 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:48.158654    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026efd88a94b6a2b6b5cec91b23e0891a54175126ba444ffede798b542f2635b\": container with ID starting with 026efd88a94b6a2b6b5cec91b23e0891a54175126ba444ffede798b542f2635b not found: ID does not exist" containerID="026efd88a94b6a2b6b5cec91b23e0891a54175126ba444ffede798b542f2635b"
Oct 29 22:14:48.158802 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:48.158694    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026efd88a94b6a2b6b5cec91b23e0891a54175126ba444ffede798b542f2635b"} err="failed to get container status \"026efd88a94b6a2b6b5cec91b23e0891a54175126ba444ffede798b542f2635b\": rpc error: code = NotFound desc = could not find container \"026efd88a94b6a2b6b5cec91b23e0891a54175126ba444ffede798b542f2635b\": container with ID starting with 026efd88a94b6a2b6b5cec91b23e0891a54175126ba444ffede798b542f2635b not found: ID does not exist"
Oct 29 22:14:48.158802 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:48.158718    4550 scope.go:122] "RemoveContainer" containerID="3b1a1e2579e8bf8fe938ff45559dae6bb7a27cd5f164817a07c4da911fa170cf"
Oct 29 22:14:48.159020 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:14:48.159000    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1a1e2579e8bf8fe938ff45559dae6bb7a27cd5f164817a07c4da911fa170cf\": container with ID starting with 3b1a1e2579e8bf8fe938ff45559dae6bb7a27cd5f164817a07c4da911fa170cf not found: ID does not exist" containerID="3b1a1e2579e8bf8fe938ff45559dae6bb7a27cd5f164817a07c4da911fa170cf"
Oct 29 22:14:48.159103 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:48.159029    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1a1e2579e8bf8fe938ff45559dae6bb7a27cd5f164817a07c4da911fa170cf"} err="failed to get container status \"3b1a1e2579e8bf8fe938ff45559dae6bb7a27cd5f164817a07c4da911fa170cf\": rpc error: code = NotFound desc = could not find container \"3b1a1e2579e8bf8fe938ff45559dae6bb7a27cd5f164817a07c4da911fa170cf\": container with ID starting with 3b1a1e2579e8bf8fe938ff45559dae6bb7a27cd5f164817a07c4da911fa170cf not found: ID does not exist"
Oct 29 22:14:48.199874 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:48.197650    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"]
Oct 29 22:14:48.217578 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:48.217547    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-lhh5p"]
Oct 29 22:14:48.412615 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:48.412579    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:49.729761 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:49.729730    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:49.830404 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:49.830370    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"]
Oct 29 22:14:49.830924 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:49.830847    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6" podUID="11f45a3b-d463-4335-b8f5-e43831a38ceb" containerName="payload-pre-processing" containerID="cri-o://2f26ba83c235b86747d7f1eb4f3ba6cfde97c0b647e2055fe178203196b1321f" gracePeriod=30
Oct 29 22:14:49.906294 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:49.906267    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:49.992059 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:49.991986    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b27b23e3-07bd-4531-9d64-73d1b66de485" path="/var/lib/kubelet/pods/b27b23e3-07bd-4531-9d64-73d1b66de485/volumes"
Oct 29 22:14:50.026818 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:50.026786    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:50.222799 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:50.218876    4550 generic.go:356] "Generic (PLEG): container finished" podID="11f45a3b-d463-4335-b8f5-e43831a38ceb" containerID="2f26ba83c235b86747d7f1eb4f3ba6cfde97c0b647e2055fe178203196b1321f" exitCode=0
Oct 29 22:14:50.222799 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:50.219079    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6" event={"ID":"11f45a3b-d463-4335-b8f5-e43831a38ceb","Type":"ContainerDied","Data":"2f26ba83c235b86747d7f1eb4f3ba6cfde97c0b647e2055fe178203196b1321f"}
Oct 29 22:14:50.232829 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:50.232806    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"
Oct 29 22:14:50.273755 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:50.273675    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f45a3b-d463-4335-b8f5-e43831a38ceb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f45a3b-d463-4335-b8f5-e43831a38ceb-plugins-config-volume\") pod \"11f45a3b-d463-4335-b8f5-e43831a38ceb\" (UID: \"11f45a3b-d463-4335-b8f5-e43831a38ceb\") "
Oct 29 22:14:50.273938 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:50.273783    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f45a3b-d463-4335-b8f5-e43831a38ceb-kube-api-access-p74kd\" (UniqueName: \"kubernetes.io/projected/11f45a3b-d463-4335-b8f5-e43831a38ceb-kube-api-access-p74kd\") pod \"11f45a3b-d463-4335-b8f5-e43831a38ceb\" (UID: \"11f45a3b-d463-4335-b8f5-e43831a38ceb\") "
Oct 29 22:14:50.274491 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:50.274061    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f45a3b-d463-4335-b8f5-e43831a38ceb-plugins-config-volume" pod "11f45a3b-d463-4335-b8f5-e43831a38ceb" (UID: "11f45a3b-d463-4335-b8f5-e43831a38ceb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:50.274629 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:50.274516    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f45a3b-d463-4335-b8f5-e43831a38ceb-plugins-config-volume\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:50.276415 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:50.276385    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f45a3b-d463-4335-b8f5-e43831a38ceb-kube-api-access-p74kd" pod "11f45a3b-d463-4335-b8f5-e43831a38ceb" (UID: "11f45a3b-d463-4335-b8f5-e43831a38ceb"). InnerVolumeSpecName "kube-api-access-p74kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:50.340004 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:50.339974    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:50.375626 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:50.375597    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p74kd\" (UniqueName: \"kubernetes.io/projected/11f45a3b-d463-4335-b8f5-e43831a38ceb-kube-api-access-p74kd\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:14:50.462788 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:50.462749    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:50.586630 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:50.586599    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:50.942350 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:50.942275    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:51.226444 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:51.226373    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"
Oct 29 22:14:51.226444 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:51.226412    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6" event={"ID":"11f45a3b-d463-4335-b8f5-e43831a38ceb","Type":"ContainerDied","Data":"7afb834208f284131d731ff80bff77a154d4dbc3fadfd8714beb8db1249aeaa7"}
Oct 29 22:14:51.226659 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:51.226482    4550 scope.go:122] "RemoveContainer" containerID="2f26ba83c235b86747d7f1eb4f3ba6cfde97c0b647e2055fe178203196b1321f"
Oct 29 22:14:51.279350 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:51.279320    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"]
Oct 29 22:14:51.291394 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:51.291364    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-9ddf6b767-dlrq6"]
Oct 29 22:14:51.988736 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:51.988707    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f45a3b-d463-4335-b8f5-e43831a38ceb" path="/var/lib/kubelet/pods/11f45a3b-d463-4335-b8f5-e43831a38ceb/volumes"
Oct 29 22:14:52.260320 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:52.259452    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:53.732084 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:53.732047    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:53.844850 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:53.844815    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:55.993276 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:55.993240    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:56.105498 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:56.105466    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:57.634810 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:57.634770    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:59.239499 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:14:59.239466    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:01.446932 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:01.446874    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:02.944044 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:02.942361    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" containerID="cri-o://77c0db2ce0493dce7314d756465856372e7f0dc2e7c68e4490e7cd5d4c83f89b" gracePeriod=45
Oct 29 22:15:02.966255 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:02.966070    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" containerID="cri-o://3815e3e8034638f89db5397b31ff7d098d778f110ac796b6c24806ccdd49e40b" gracePeriod=45
Oct 29 22:15:03.265349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:03.262375    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": EOF"
Oct 29 22:15:03.265349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:03.262860    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": dial tcp 10.133.0.43:8000: connect: connection refused"
Oct 29 22:15:03.265349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:03.262956    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:15:03.353172 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:03.352253    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": dial tcp 10.133.0.44:8000: connect: connection refused"
Oct 29 22:15:03.549179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:03.545516    4550 generic.go:356] "Generic (PLEG): container finished" podID="280967ca-516c-40d1-891f-7c3edb621486" containerID="77c0db2ce0493dce7314d756465856372e7f0dc2e7c68e4490e7cd5d4c83f89b" exitCode=0
Oct 29 22:15:03.549179 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:03.545581    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" event={"ID":"280967ca-516c-40d1-891f-7c3edb621486","Type":"ContainerDied","Data":"77c0db2ce0493dce7314d756465856372e7f0dc2e7c68e4490e7cd5d4c83f89b"}
Oct 29 22:15:03.921175 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:03.920292    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:04.282156 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.281850    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:15:04.358417 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.358379    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-tmp-dir\") pod \"1021a25d-cebc-4bad-965d-6afc5bad66f9\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") "
Oct 29 22:15:04.358588 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.358562    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-model-cache\") pod \"1021a25d-cebc-4bad-965d-6afc5bad66f9\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") "
Oct 29 22:15:04.359078 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.358799    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-dshm\") pod \"1021a25d-cebc-4bad-965d-6afc5bad66f9\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") "
Oct 29 22:15:04.359078 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.358874    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-home\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-home\") pod \"1021a25d-cebc-4bad-965d-6afc5bad66f9\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") "
Oct 29 22:15:04.359078 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.358911    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1021a25d-cebc-4bad-965d-6afc5bad66f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/1021a25d-cebc-4bad-965d-6afc5bad66f9-tls-certs\") pod \"1021a25d-cebc-4bad-965d-6afc5bad66f9\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") "
Oct 29 22:15:04.359078 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.358973    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1021a25d-cebc-4bad-965d-6afc5bad66f9-kube-api-access-p5mbb\" (UniqueName: \"kubernetes.io/projected/1021a25d-cebc-4bad-965d-6afc5bad66f9-kube-api-access-p5mbb\") pod \"1021a25d-cebc-4bad-965d-6afc5bad66f9\" (UID: \"1021a25d-cebc-4bad-965d-6afc5bad66f9\") "
Oct 29 22:15:04.360968 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.360786    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-tmp-dir" pod "1021a25d-cebc-4bad-965d-6afc5bad66f9" (UID: "1021a25d-cebc-4bad-965d-6afc5bad66f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:04.360968 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.360882    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-model-cache" pod "1021a25d-cebc-4bad-965d-6afc5bad66f9" (UID: "1021a25d-cebc-4bad-965d-6afc5bad66f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:04.360968 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.360940    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-home" pod "1021a25d-cebc-4bad-965d-6afc5bad66f9" (UID: "1021a25d-cebc-4bad-965d-6afc5bad66f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:04.364012 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.363966    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-dshm" pod "1021a25d-cebc-4bad-965d-6afc5bad66f9" (UID: "1021a25d-cebc-4bad-965d-6afc5bad66f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:04.373744 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.373286    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1021a25d-cebc-4bad-965d-6afc5bad66f9-tls-certs" pod "1021a25d-cebc-4bad-965d-6afc5bad66f9" (UID: "1021a25d-cebc-4bad-965d-6afc5bad66f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:04.375479 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.375326    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1021a25d-cebc-4bad-965d-6afc5bad66f9-kube-api-access-p5mbb" pod "1021a25d-cebc-4bad-965d-6afc5bad66f9" (UID: "1021a25d-cebc-4bad-965d-6afc5bad66f9"). InnerVolumeSpecName "kube-api-access-p5mbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:04.385528 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.385481    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:15:04.462866 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.460167    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280967ca-516c-40d1-891f-7c3edb621486-kube-api-access-tz72m\" (UniqueName: \"kubernetes.io/projected/280967ca-516c-40d1-891f-7c3edb621486-kube-api-access-tz72m\") pod \"280967ca-516c-40d1-891f-7c3edb621486\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") "
Oct 29 22:15:04.463612 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.463185    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-dshm\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-dshm\") pod \"280967ca-516c-40d1-891f-7c3edb621486\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") "
Oct 29 22:15:04.463872 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.463851    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-home\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-home\") pod \"280967ca-516c-40d1-891f-7c3edb621486\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") "
Oct 29 22:15:04.464381 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.464256    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-home" pod "280967ca-516c-40d1-891f-7c3edb621486" (UID: "280967ca-516c-40d1-891f-7c3edb621486"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:04.464594 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.464574    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-model-cache\") pod \"280967ca-516c-40d1-891f-7c3edb621486\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") "
Oct 29 22:15:04.465034 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.464919    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-model-cache" pod "280967ca-516c-40d1-891f-7c3edb621486" (UID: "280967ca-516c-40d1-891f-7c3edb621486"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:04.465211 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.465195    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/280967ca-516c-40d1-891f-7c3edb621486-tls-certs\" (UniqueName: \"kubernetes.io/secret/280967ca-516c-40d1-891f-7c3edb621486-tls-certs\") pod \"280967ca-516c-40d1-891f-7c3edb621486\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") "
Oct 29 22:15:04.466173 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.465810    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-tmp-dir\") pod \"280967ca-516c-40d1-891f-7c3edb621486\" (UID: \"280967ca-516c-40d1-891f-7c3edb621486\") "
Oct 29 22:15:04.466406 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.466064    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-tmp-dir" pod "280967ca-516c-40d1-891f-7c3edb621486" (UID: "280967ca-516c-40d1-891f-7c3edb621486"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:04.467450 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.467419    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-dshm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.467587 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.467573    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-tmp-dir\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.467704 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.467693    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-home\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.467808 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.467786    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1021a25d-cebc-4bad-965d-6afc5bad66f9-tls-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.467926 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.467912    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5mbb\" (UniqueName: \"kubernetes.io/projected/1021a25d-cebc-4bad-965d-6afc5bad66f9-kube-api-access-p5mbb\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.468020 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.468007    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-tmp-dir\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.468103 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.468093    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-home\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.468262 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.468249    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1021a25d-cebc-4bad-965d-6afc5bad66f9-model-cache\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.468763 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.468748    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-model-cache\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.468907 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.468702    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-dshm" pod "280967ca-516c-40d1-891f-7c3edb621486" (UID: "280967ca-516c-40d1-891f-7c3edb621486"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:04.469027 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.468742    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280967ca-516c-40d1-891f-7c3edb621486-kube-api-access-tz72m" pod "280967ca-516c-40d1-891f-7c3edb621486" (UID: "280967ca-516c-40d1-891f-7c3edb621486"). InnerVolumeSpecName "kube-api-access-tz72m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:04.469360 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.469331    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/280967ca-516c-40d1-891f-7c3edb621486-tls-certs" pod "280967ca-516c-40d1-891f-7c3edb621486" (UID: "280967ca-516c-40d1-891f-7c3edb621486"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:04.554269 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.554114    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"
Oct 29 22:15:04.554269 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.554177    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" event={"ID":"280967ca-516c-40d1-891f-7c3edb621486","Type":"ContainerDied","Data":"97912b1ee22297ed53851b26f56e9f4e283eb7930fbef51eab462d7f62f78615"}
Oct 29 22:15:04.554269 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.554229    4550 scope.go:122] "RemoveContainer" containerID="77c0db2ce0493dce7314d756465856372e7f0dc2e7c68e4490e7cd5d4c83f89b"
Oct 29 22:15:04.561156 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.561112    4550 generic.go:356] "Generic (PLEG): container finished" podID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerID="3815e3e8034638f89db5397b31ff7d098d778f110ac796b6c24806ccdd49e40b" exitCode=0
Oct 29 22:15:04.561261 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.561240    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"
Oct 29 22:15:04.561349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.561328    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" event={"ID":"1021a25d-cebc-4bad-965d-6afc5bad66f9","Type":"ContainerDied","Data":"3815e3e8034638f89db5397b31ff7d098d778f110ac796b6c24806ccdd49e40b"}
Oct 29 22:15:04.561410 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.561375    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c" event={"ID":"1021a25d-cebc-4bad-965d-6afc5bad66f9","Type":"ContainerDied","Data":"4d7bb4b6938344b2d996cb6ee38c8323402ffbdde74a73a099ad66f8d65f5b82"}
Oct 29 22:15:04.570001 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.569966    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz72m\" (UniqueName: \"kubernetes.io/projected/280967ca-516c-40d1-891f-7c3edb621486-kube-api-access-tz72m\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.570218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.570003    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/280967ca-516c-40d1-891f-7c3edb621486-dshm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.570218 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.570020    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/280967ca-516c-40d1-891f-7c3edb621486-tls-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.571669 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.571557    4550 scope.go:122] "RemoveContainer" containerID="3815e3e8034638f89db5397b31ff7d098d778f110ac796b6c24806ccdd49e40b"
Oct 29 22:15:04.581744 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.581724    4550 scope.go:122] "RemoveContainer" containerID="3815e3e8034638f89db5397b31ff7d098d778f110ac796b6c24806ccdd49e40b"
Oct 29 22:15:04.582058 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:15:04.582037    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3815e3e8034638f89db5397b31ff7d098d778f110ac796b6c24806ccdd49e40b\": container with ID starting with 3815e3e8034638f89db5397b31ff7d098d778f110ac796b6c24806ccdd49e40b not found: ID does not exist" containerID="3815e3e8034638f89db5397b31ff7d098d778f110ac796b6c24806ccdd49e40b"
Oct 29 22:15:04.582200 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.582066    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3815e3e8034638f89db5397b31ff7d098d778f110ac796b6c24806ccdd49e40b"} err="failed to get container status \"3815e3e8034638f89db5397b31ff7d098d778f110ac796b6c24806ccdd49e40b\": rpc error: code = NotFound desc = could not find container \"3815e3e8034638f89db5397b31ff7d098d778f110ac796b6c24806ccdd49e40b\": container with ID starting with 3815e3e8034638f89db5397b31ff7d098d778f110ac796b6c24806ccdd49e40b not found: ID does not exist"
Oct 29 22:15:04.620713 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.620681    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"]
Oct 29 22:15:04.629510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.629477    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t"]
Oct 29 22:15:04.663555 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.663517    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"]
Oct 29 22:15:04.666722 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:04.666687    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/simulator-w0-e4e24e-kserve-6f76df77d4-8fh4c"]
Oct 29 22:15:05.351001 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:05.350959    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/premium-w0-e4e24e-kserve-56954b7d4f-7vf8t" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:15:05.861064 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:05.861023    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:05.993921 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:05.993887    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" path="/var/lib/kubelet/pods/1021a25d-cebc-4bad-965d-6afc5bad66f9/volumes"
Oct 29 22:15:05.994358 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:05.994342    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="280967ca-516c-40d1-891f-7c3edb621486" path="/var/lib/kubelet/pods/280967ca-516c-40d1-891f-7c3edb621486/volumes"
Oct 29 22:15:07.601759 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:07.601724    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"]
Oct 29 22:15:07.602287 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:07.601967    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc" podUID="a3b59718-adb6-437a-8c5d-3003d93b5203" containerName="maas-api" containerID="cri-o://a9fd0940145c0102548abf2f08c2f56edd9ddf7789449e0494275813404b88ac" gracePeriod=30
Oct 29 22:15:07.656682 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:07.656635    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:07.994164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:07.991374    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 22:15:08.111886 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.111832    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-metrics-certs\") pod \"a3b59718-adb6-437a-8c5d-3003d93b5203\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") "
Oct 29 22:15:08.112059 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.111987    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b59718-adb6-437a-8c5d-3003d93b5203-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b59718-adb6-437a-8c5d-3003d93b5203-openshift-service-ca\") pod \"a3b59718-adb6-437a-8c5d-3003d93b5203\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") "
Oct 29 22:15:08.112059 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.112040    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-maas-api-tls\") pod \"a3b59718-adb6-437a-8c5d-3003d93b5203\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") "
Oct 29 22:15:08.112136 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.112092    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b59718-adb6-437a-8c5d-3003d93b5203-kube-api-access-9nsqp\" (UniqueName: \"kubernetes.io/projected/a3b59718-adb6-437a-8c5d-3003d93b5203-kube-api-access-9nsqp\") pod \"a3b59718-adb6-437a-8c5d-3003d93b5203\" (UID: \"a3b59718-adb6-437a-8c5d-3003d93b5203\") "
Oct 29 22:15:08.114161 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.114125    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b59718-adb6-437a-8c5d-3003d93b5203-openshift-service-ca" pod "a3b59718-adb6-437a-8c5d-3003d93b5203" (UID: "a3b59718-adb6-437a-8c5d-3003d93b5203"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:08.119703 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.119644    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b59718-adb6-437a-8c5d-3003d93b5203-kube-api-access-9nsqp" pod "a3b59718-adb6-437a-8c5d-3003d93b5203" (UID: "a3b59718-adb6-437a-8c5d-3003d93b5203"). InnerVolumeSpecName "kube-api-access-9nsqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:08.123015 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.122608    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-metrics-certs" pod "a3b59718-adb6-437a-8c5d-3003d93b5203" (UID: "a3b59718-adb6-437a-8c5d-3003d93b5203"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:08.124349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.124316    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-maas-api-tls" pod "a3b59718-adb6-437a-8c5d-3003d93b5203" (UID: "a3b59718-adb6-437a-8c5d-3003d93b5203"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:08.213897 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.213812    4550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-metrics-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:08.213897 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.213842    4550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b59718-adb6-437a-8c5d-3003d93b5203-openshift-service-ca\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:08.213897 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.213852    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b59718-adb6-437a-8c5d-3003d93b5203-maas-api-tls\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:08.213897 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.213861    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nsqp\" (UniqueName: \"kubernetes.io/projected/a3b59718-adb6-437a-8c5d-3003d93b5203-kube-api-access-9nsqp\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:08.258973 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.258938    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:08.373112 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.373081    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:08.491963 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.491789    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:08.601263 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.601233    4550 generic.go:356] "Generic (PLEG): container finished" podID="a3b59718-adb6-437a-8c5d-3003d93b5203" containerID="a9fd0940145c0102548abf2f08c2f56edd9ddf7789449e0494275813404b88ac" exitCode=0
Oct 29 22:15:08.601447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.601278    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"
Oct 29 22:15:08.601447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.601318    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc" event={"ID":"a3b59718-adb6-437a-8c5d-3003d93b5203","Type":"ContainerDied","Data":"a9fd0940145c0102548abf2f08c2f56edd9ddf7789449e0494275813404b88ac"}
Oct 29 22:15:08.601447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.601374    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc" event={"ID":"a3b59718-adb6-437a-8c5d-3003d93b5203","Type":"ContainerDied","Data":"306d8b182e876cb7b9f94a1d621893c040112d53403752a17b4231a654134fa6"}
Oct 29 22:15:08.601447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.601397    4550 scope.go:122] "RemoveContainer" containerID="a9fd0940145c0102548abf2f08c2f56edd9ddf7789449e0494275813404b88ac"
Oct 29 22:15:08.611988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.611831    4550 scope.go:122] "RemoveContainer" containerID="a9fd0940145c0102548abf2f08c2f56edd9ddf7789449e0494275813404b88ac"
Oct 29 22:15:08.612246 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:15:08.612136    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9fd0940145c0102548abf2f08c2f56edd9ddf7789449e0494275813404b88ac\": container with ID starting with a9fd0940145c0102548abf2f08c2f56edd9ddf7789449e0494275813404b88ac not found: ID does not exist" containerID="a9fd0940145c0102548abf2f08c2f56edd9ddf7789449e0494275813404b88ac"
Oct 29 22:15:08.612246 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.612200    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9fd0940145c0102548abf2f08c2f56edd9ddf7789449e0494275813404b88ac"} err="failed to get container status \"a9fd0940145c0102548abf2f08c2f56edd9ddf7789449e0494275813404b88ac\": rpc error: code = NotFound desc = could not find container \"a9fd0940145c0102548abf2f08c2f56edd9ddf7789449e0494275813404b88ac\": container with ID starting with a9fd0940145c0102548abf2f08c2f56edd9ddf7789449e0494275813404b88ac not found: ID does not exist"
Oct 29 22:15:08.652302 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.652263    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"]
Oct 29 22:15:08.664839 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.664805    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e4e24e-5f97c887b-p6bgc"]
Oct 29 22:15:08.907795 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:08.907764    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:09.988405 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:09.988371    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b59718-adb6-437a-8c5d-3003d93b5203" path="/var/lib/kubelet/pods/a3b59718-adb6-437a-8c5d-3003d93b5203/volumes"
Oct 29 22:15:10.107154 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:10.107116    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:11.490679 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:15:11.490643    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:15:11.949018 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:11.948985    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:14.081452 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:14.081418    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:14.427480 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:14.427402    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:15.583673 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:15.583639    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:15.733049 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:15.733015    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:15.938177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:15.938065    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"]
Oct 29 22:15:15.938631 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:15.938602    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6" podUID="5a89ba55-981b-475e-a86c-833c547e65d5" containerName="istio-proxy" containerID="cri-o://377137e92fa9ed2894a0b7a6a0c0d53b0710ecedb11b2d868728a49a99017b88" gracePeriod=30
Oct 29 22:15:15.946288 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:15.946259    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"]
Oct 29 22:15:15.946491 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:15.946465    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628" podUID="2fbcc15a-7f6f-4742-aabb-916bb609889a" containerName="istio-proxy" containerID="cri-o://9f4e64e0dc5a927dafc93db5a58cfdd5c7d60552a85702a8b060f93e201b316d" gracePeriod=30
Oct 29 22:15:15.948763 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:15.948738    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"]
Oct 29 22:15:15.948902 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:15.948880    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw" podUID="df402fae-12c4-4ed9-9fb1-226a51f66c18" containerName="istio-proxy" containerID="cri-o://a14c5909e303534e631514d330176e6344fc2cb135c2feb6c3b192c6bb79af6f" gracePeriod=30
Oct 29 22:15:15.987167 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:15.983057    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:16.105556 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:16.105526    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:17.539156 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:17.539057    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:19.649780 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:19.649745    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:21.218417 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.218336    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:21.229365 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.229342    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 22:15:21.317647 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.317612    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 22:15:21.321938 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.321569    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 22:15:21.390859 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.390829    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-envoy\") pod \"2fbcc15a-7f6f-4742-aabb-916bb609889a\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") "
Oct 29 22:15:21.391041 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.390878    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a89ba55-981b-475e-a86c-833c547e65d5-istio-token\" (UniqueName: \"kubernetes.io/projected/5a89ba55-981b-475e-a86c-833c547e65d5-istio-token\") pod \"5a89ba55-981b-475e-a86c-833c547e65d5\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") "
Oct 29 22:15:21.391041 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.390910    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-workload-socket\") pod \"5a89ba55-981b-475e-a86c-833c547e65d5\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") "
Oct 29 22:15:21.391041 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.390933    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-workload-certs\") pod \"5a89ba55-981b-475e-a86c-833c547e65d5\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") "
Oct 29 22:15:21.391041 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.390951    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a89ba55-981b-475e-a86c-833c547e65d5-kube-api-access-vkj5d\" (UniqueName: \"kubernetes.io/projected/5a89ba55-981b-475e-a86c-833c547e65d5-kube-api-access-vkj5d\") pod \"5a89ba55-981b-475e-a86c-833c547e65d5\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") "
Oct 29 22:15:21.391041 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.390998    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-data\") pod \"2fbcc15a-7f6f-4742-aabb-916bb609889a\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") "
Oct 29 22:15:21.391041 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391034    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-credential-socket\") pod \"5a89ba55-981b-475e-a86c-833c547e65d5\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") "
Oct 29 22:15:21.391450 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391074    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-workload-socket\") pod \"2fbcc15a-7f6f-4742-aabb-916bb609889a\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") "
Oct 29 22:15:21.391450 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391132    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fbcc15a-7f6f-4742-aabb-916bb609889a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fbcc15a-7f6f-4742-aabb-916bb609889a-istiod-ca-cert\") pod \"2fbcc15a-7f6f-4742-aabb-916bb609889a\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") "
Oct 29 22:15:21.391450 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391230    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-workload-certs\") pod \"2fbcc15a-7f6f-4742-aabb-916bb609889a\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") "
Oct 29 22:15:21.391450 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391262    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-istio-data\") pod \"5a89ba55-981b-475e-a86c-833c547e65d5\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") "
Oct 29 22:15:21.391450 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391342    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-credential-socket\") pod \"2fbcc15a-7f6f-4742-aabb-916bb609889a\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") "
Oct 29 22:15:21.391450 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391382    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-podinfo\") pod \"2fbcc15a-7f6f-4742-aabb-916bb609889a\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") "
Oct 29 22:15:21.391450 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391412    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fbcc15a-7f6f-4742-aabb-916bb609889a-kube-api-access-5gppr\" (UniqueName: \"kubernetes.io/projected/2fbcc15a-7f6f-4742-aabb-916bb609889a-kube-api-access-5gppr\") pod \"2fbcc15a-7f6f-4742-aabb-916bb609889a\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") "
Oct 29 22:15:21.391450 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391259    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-workload-socket" pod "5a89ba55-981b-475e-a86c-833c547e65d5" (UID: "5a89ba55-981b-475e-a86c-833c547e65d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.391878 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391374    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-credential-socket" pod "5a89ba55-981b-475e-a86c-833c547e65d5" (UID: "5a89ba55-981b-475e-a86c-833c547e65d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.391878 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391426    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-data" pod "2fbcc15a-7f6f-4742-aabb-916bb609889a" (UID: "2fbcc15a-7f6f-4742-aabb-916bb609889a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.391878 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391564    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-workload-certs" pod "5a89ba55-981b-475e-a86c-833c547e65d5" (UID: "5a89ba55-981b-475e-a86c-833c547e65d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.391878 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391572    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-workload-socket" pod "2fbcc15a-7f6f-4742-aabb-916bb609889a" (UID: "2fbcc15a-7f6f-4742-aabb-916bb609889a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.391878 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391686    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-workload-certs" pod "2fbcc15a-7f6f-4742-aabb-916bb609889a" (UID: "2fbcc15a-7f6f-4742-aabb-916bb609889a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.391878 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391465    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a89ba55-981b-475e-a86c-833c547e65d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a89ba55-981b-475e-a86c-833c547e65d5-istiod-ca-cert\") pod \"5a89ba55-981b-475e-a86c-833c547e65d5\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") "
Oct 29 22:15:21.391878 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391707    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fbcc15a-7f6f-4742-aabb-916bb609889a-istiod-ca-cert" pod "2fbcc15a-7f6f-4742-aabb-916bb609889a" (UID: "2fbcc15a-7f6f-4742-aabb-916bb609889a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:21.391878 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391759    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a89ba55-981b-475e-a86c-833c547e65d5-istiod-ca-cert" pod "5a89ba55-981b-475e-a86c-833c547e65d5" (UID: "5a89ba55-981b-475e-a86c-833c547e65d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:21.391878 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391768    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-token\" (UniqueName: \"kubernetes.io/projected/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-token\") pod \"2fbcc15a-7f6f-4742-aabb-916bb609889a\" (UID: \"2fbcc15a-7f6f-4742-aabb-916bb609889a\") "
Oct 29 22:15:21.391878 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391814    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a89ba55-981b-475e-a86c-833c547e65d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a89ba55-981b-475e-a86c-833c547e65d5-istio-podinfo\") pod \"5a89ba55-981b-475e-a86c-833c547e65d5\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") "
Oct 29 22:15:21.392466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391902    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-istio-data" pod "5a89ba55-981b-475e-a86c-833c547e65d5" (UID: "5a89ba55-981b-475e-a86c-833c547e65d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.392466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.391924    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-istio-envoy\") pod \"5a89ba55-981b-475e-a86c-833c547e65d5\" (UID: \"5a89ba55-981b-475e-a86c-833c547e65d5\") "
Oct 29 22:15:21.392466 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.392003    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-credential-socket" pod "2fbcc15a-7f6f-4742-aabb-916bb609889a" (UID: "2fbcc15a-7f6f-4742-aabb-916bb609889a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.392639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.392463    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.392639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.392486    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.392639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.392500    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.392639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.392515    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.392639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.392528    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.392639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.392541    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fbcc15a-7f6f-4742-aabb-916bb609889a-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.392639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.392555    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.392639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.392567    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.392639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.392579    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.392639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.392592    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a89ba55-981b-475e-a86c-833c547e65d5-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.394546 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.394520    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fbcc15a-7f6f-4742-aabb-916bb609889a-kube-api-access-5gppr" pod "2fbcc15a-7f6f-4742-aabb-916bb609889a" (UID: "2fbcc15a-7f6f-4742-aabb-916bb609889a"). InnerVolumeSpecName "kube-api-access-5gppr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.394765 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.394741    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-podinfo" pod "2fbcc15a-7f6f-4742-aabb-916bb609889a" (UID: "2fbcc15a-7f6f-4742-aabb-916bb609889a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:21.395078 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.395053    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-envoy" pod "2fbcc15a-7f6f-4742-aabb-916bb609889a" (UID: "2fbcc15a-7f6f-4742-aabb-916bb609889a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.395481 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.395458    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-istio-envoy" pod "5a89ba55-981b-475e-a86c-833c547e65d5" (UID: "5a89ba55-981b-475e-a86c-833c547e65d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.396754 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.396716    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a89ba55-981b-475e-a86c-833c547e65d5-istio-podinfo" pod "5a89ba55-981b-475e-a86c-833c547e65d5" (UID: "5a89ba55-981b-475e-a86c-833c547e65d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:21.397660 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.397632    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a89ba55-981b-475e-a86c-833c547e65d5-kube-api-access-vkj5d" pod "5a89ba55-981b-475e-a86c-833c547e65d5" (UID: "5a89ba55-981b-475e-a86c-833c547e65d5"). InnerVolumeSpecName "kube-api-access-vkj5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.397841 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.397819    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a89ba55-981b-475e-a86c-833c547e65d5-istio-token" pod "5a89ba55-981b-475e-a86c-833c547e65d5" (UID: "5a89ba55-981b-475e-a86c-833c547e65d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.398202 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.398121    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-token" pod "2fbcc15a-7f6f-4742-aabb-916bb609889a" (UID: "2fbcc15a-7f6f-4742-aabb-916bb609889a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.493343 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.493256    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-workload-certs\") pod \"df402fae-12c4-4ed9-9fb1-226a51f66c18\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") "
Oct 29 22:15:21.493343 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.493308    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-data\") pod \"df402fae-12c4-4ed9-9fb1-226a51f66c18\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") "
Oct 29 22:15:21.493557 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.493347    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-podinfo\") pod \"df402fae-12c4-4ed9-9fb1-226a51f66c18\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") "
Oct 29 22:15:21.493557 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.493388    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-envoy\") pod \"df402fae-12c4-4ed9-9fb1-226a51f66c18\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") "
Oct 29 22:15:21.493557 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.493414    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-credential-socket\") pod \"df402fae-12c4-4ed9-9fb1-226a51f66c18\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") "
Oct 29 22:15:21.493557 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.493477    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df402fae-12c4-4ed9-9fb1-226a51f66c18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df402fae-12c4-4ed9-9fb1-226a51f66c18-istiod-ca-cert\") pod \"df402fae-12c4-4ed9-9fb1-226a51f66c18\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") "
Oct 29 22:15:21.493557 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.493525    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-token\" (UniqueName: \"kubernetes.io/projected/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-token\") pod \"df402fae-12c4-4ed9-9fb1-226a51f66c18\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") "
Oct 29 22:15:21.493798 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.493566    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df402fae-12c4-4ed9-9fb1-226a51f66c18-kube-api-access-df7dk\" (UniqueName: \"kubernetes.io/projected/df402fae-12c4-4ed9-9fb1-226a51f66c18-kube-api-access-df7dk\") pod \"df402fae-12c4-4ed9-9fb1-226a51f66c18\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") "
Oct 29 22:15:21.493798 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.493580    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-workload-certs" pod "df402fae-12c4-4ed9-9fb1-226a51f66c18" (UID: "df402fae-12c4-4ed9-9fb1-226a51f66c18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.493798 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.493589    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-data" pod "df402fae-12c4-4ed9-9fb1-226a51f66c18" (UID: "df402fae-12c4-4ed9-9fb1-226a51f66c18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.493798 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.493675    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-workload-socket\") pod \"df402fae-12c4-4ed9-9fb1-226a51f66c18\" (UID: \"df402fae-12c4-4ed9-9fb1-226a51f66c18\") "
Oct 29 22:15:21.493987 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.493797    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-credential-socket" pod "df402fae-12c4-4ed9-9fb1-226a51f66c18" (UID: "df402fae-12c4-4ed9-9fb1-226a51f66c18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.494540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.494103    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.494540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.494126    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.494540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.494158    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gppr\" (UniqueName: \"kubernetes.io/projected/2fbcc15a-7f6f-4742-aabb-916bb609889a-kube-api-access-5gppr\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.494540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.494174    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.494540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.494187    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.494540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.494199    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.494540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.494214    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a89ba55-981b-475e-a86c-833c547e65d5-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.494540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.494227    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a89ba55-981b-475e-a86c-833c547e65d5-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.494540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.494239    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fbcc15a-7f6f-4742-aabb-916bb609889a-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.494540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.494251    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a89ba55-981b-475e-a86c-833c547e65d5-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.494540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.494264    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkj5d\" (UniqueName: \"kubernetes.io/projected/5a89ba55-981b-475e-a86c-833c547e65d5-kube-api-access-vkj5d\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.494540 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.494466    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-workload-socket" pod "df402fae-12c4-4ed9-9fb1-226a51f66c18" (UID: "df402fae-12c4-4ed9-9fb1-226a51f66c18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.495352 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.494656    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df402fae-12c4-4ed9-9fb1-226a51f66c18-istiod-ca-cert" pod "df402fae-12c4-4ed9-9fb1-226a51f66c18" (UID: "df402fae-12c4-4ed9-9fb1-226a51f66c18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:21.496290 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.496253    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-envoy" pod "df402fae-12c4-4ed9-9fb1-226a51f66c18" (UID: "df402fae-12c4-4ed9-9fb1-226a51f66c18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.496290 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.496262    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-podinfo" pod "df402fae-12c4-4ed9-9fb1-226a51f66c18" (UID: "df402fae-12c4-4ed9-9fb1-226a51f66c18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:21.496478 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.496462    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-token" pod "df402fae-12c4-4ed9-9fb1-226a51f66c18" (UID: "df402fae-12c4-4ed9-9fb1-226a51f66c18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.496578 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.496561    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df402fae-12c4-4ed9-9fb1-226a51f66c18-kube-api-access-df7dk" pod "df402fae-12c4-4ed9-9fb1-226a51f66c18" (UID: "df402fae-12c4-4ed9-9fb1-226a51f66c18"). InnerVolumeSpecName "kube-api-access-df7dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.607117 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.607084    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.607117 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.607115    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df7dk\" (UniqueName: \"kubernetes.io/projected/df402fae-12c4-4ed9-9fb1-226a51f66c18-kube-api-access-df7dk\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.607117 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.607125    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.607390 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.607136    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.607390 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.607161    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df402fae-12c4-4ed9-9fb1-226a51f66c18-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.607390 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.607172    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df402fae-12c4-4ed9-9fb1-226a51f66c18-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.689364 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.689324    4550 generic.go:356] "Generic (PLEG): container finished" podID="5a89ba55-981b-475e-a86c-833c547e65d5" containerID="377137e92fa9ed2894a0b7a6a0c0d53b0710ecedb11b2d868728a49a99017b88" exitCode=0
Oct 29 22:15:21.689539 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.689380    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"
Oct 29 22:15:21.689539 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.689401    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6" event={"ID":"5a89ba55-981b-475e-a86c-833c547e65d5","Type":"ContainerDied","Data":"377137e92fa9ed2894a0b7a6a0c0d53b0710ecedb11b2d868728a49a99017b88"}
Oct 29 22:15:21.689539 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.689489    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6" event={"ID":"5a89ba55-981b-475e-a86c-833c547e65d5","Type":"ContainerDied","Data":"05e30f1e115ac6f46cedffa4711a8777d71976bc84478f2cb33a06de6b7eb519"}
Oct 29 22:15:21.689539 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.689521    4550 scope.go:122] "RemoveContainer" containerID="377137e92fa9ed2894a0b7a6a0c0d53b0710ecedb11b2d868728a49a99017b88"
Oct 29 22:15:21.690918 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.690893    4550 generic.go:356] "Generic (PLEG): container finished" podID="df402fae-12c4-4ed9-9fb1-226a51f66c18" containerID="a14c5909e303534e631514d330176e6344fc2cb135c2feb6c3b192c6bb79af6f" exitCode=0
Oct 29 22:15:21.691102 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.690929    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"
Oct 29 22:15:21.691102 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.690970    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw" event={"ID":"df402fae-12c4-4ed9-9fb1-226a51f66c18","Type":"ContainerDied","Data":"a14c5909e303534e631514d330176e6344fc2cb135c2feb6c3b192c6bb79af6f"}
Oct 29 22:15:21.691102 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.691024    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw" event={"ID":"df402fae-12c4-4ed9-9fb1-226a51f66c18","Type":"ContainerDied","Data":"b568132abc1a2691f14d6407b41c52c3f1444d8229568072026216ffe9344d3d"}
Oct 29 22:15:21.692443 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.692422    4550 generic.go:356] "Generic (PLEG): container finished" podID="2fbcc15a-7f6f-4742-aabb-916bb609889a" containerID="9f4e64e0dc5a927dafc93db5a58cfdd5c7d60552a85702a8b060f93e201b316d" exitCode=0
Oct 29 22:15:21.692576 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.692471    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"
Oct 29 22:15:21.692576 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.692495    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628" event={"ID":"2fbcc15a-7f6f-4742-aabb-916bb609889a","Type":"ContainerDied","Data":"9f4e64e0dc5a927dafc93db5a58cfdd5c7d60552a85702a8b060f93e201b316d"}
Oct 29 22:15:21.692576 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.692533    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628" event={"ID":"2fbcc15a-7f6f-4742-aabb-916bb609889a","Type":"ContainerDied","Data":"d46cc86a5bcc28f44f514a171e8f30d83a7006aaec31226e1907f415a000a83a"}
Oct 29 22:15:21.699350 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.699333    4550 scope.go:122] "RemoveContainer" containerID="377137e92fa9ed2894a0b7a6a0c0d53b0710ecedb11b2d868728a49a99017b88"
Oct 29 22:15:21.699607 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:15:21.699576    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377137e92fa9ed2894a0b7a6a0c0d53b0710ecedb11b2d868728a49a99017b88\": container with ID starting with 377137e92fa9ed2894a0b7a6a0c0d53b0710ecedb11b2d868728a49a99017b88 not found: ID does not exist" containerID="377137e92fa9ed2894a0b7a6a0c0d53b0710ecedb11b2d868728a49a99017b88"
Oct 29 22:15:21.699739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.699611    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377137e92fa9ed2894a0b7a6a0c0d53b0710ecedb11b2d868728a49a99017b88"} err="failed to get container status \"377137e92fa9ed2894a0b7a6a0c0d53b0710ecedb11b2d868728a49a99017b88\": rpc error: code = NotFound desc = could not find container \"377137e92fa9ed2894a0b7a6a0c0d53b0710ecedb11b2d868728a49a99017b88\": container with ID starting with 377137e92fa9ed2894a0b7a6a0c0d53b0710ecedb11b2d868728a49a99017b88 not found: ID does not exist"
Oct 29 22:15:21.699739 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.699630    4550 scope.go:122] "RemoveContainer" containerID="a14c5909e303534e631514d330176e6344fc2cb135c2feb6c3b192c6bb79af6f"
Oct 29 22:15:21.709128 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.709111    4550 scope.go:122] "RemoveContainer" containerID="a14c5909e303534e631514d330176e6344fc2cb135c2feb6c3b192c6bb79af6f"
Oct 29 22:15:21.709401 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:15:21.709384    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14c5909e303534e631514d330176e6344fc2cb135c2feb6c3b192c6bb79af6f\": container with ID starting with a14c5909e303534e631514d330176e6344fc2cb135c2feb6c3b192c6bb79af6f not found: ID does not exist" containerID="a14c5909e303534e631514d330176e6344fc2cb135c2feb6c3b192c6bb79af6f"
Oct 29 22:15:21.709459 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.709407    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14c5909e303534e631514d330176e6344fc2cb135c2feb6c3b192c6bb79af6f"} err="failed to get container status \"a14c5909e303534e631514d330176e6344fc2cb135c2feb6c3b192c6bb79af6f\": rpc error: code = NotFound desc = could not find container \"a14c5909e303534e631514d330176e6344fc2cb135c2feb6c3b192c6bb79af6f\": container with ID starting with a14c5909e303534e631514d330176e6344fc2cb135c2feb6c3b192c6bb79af6f not found: ID does not exist"
Oct 29 22:15:21.709459 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.709423    4550 scope.go:122] "RemoveContainer" containerID="9f4e64e0dc5a927dafc93db5a58cfdd5c7d60552a85702a8b060f93e201b316d"
Oct 29 22:15:21.717914 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.717898    4550 scope.go:122] "RemoveContainer" containerID="9f4e64e0dc5a927dafc93db5a58cfdd5c7d60552a85702a8b060f93e201b316d"
Oct 29 22:15:21.718133 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:15:21.718114    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f4e64e0dc5a927dafc93db5a58cfdd5c7d60552a85702a8b060f93e201b316d\": container with ID starting with 9f4e64e0dc5a927dafc93db5a58cfdd5c7d60552a85702a8b060f93e201b316d not found: ID does not exist" containerID="9f4e64e0dc5a927dafc93db5a58cfdd5c7d60552a85702a8b060f93e201b316d"
Oct 29 22:15:21.718200 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.718153    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f4e64e0dc5a927dafc93db5a58cfdd5c7d60552a85702a8b060f93e201b316d"} err="failed to get container status \"9f4e64e0dc5a927dafc93db5a58cfdd5c7d60552a85702a8b060f93e201b316d\": rpc error: code = NotFound desc = could not find container \"9f4e64e0dc5a927dafc93db5a58cfdd5c7d60552a85702a8b060f93e201b316d\": container with ID starting with 9f4e64e0dc5a927dafc93db5a58cfdd5c7d60552a85702a8b060f93e201b316d not found: ID does not exist"
Oct 29 22:15:21.741092 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.741073    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"]
Oct 29 22:15:21.743068 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.743048    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-gh8x6"]
Oct 29 22:15:21.789590 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.789527    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"]
Oct 29 22:15:21.793996 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.793976    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-bcqmw"]
Oct 29 22:15:21.826629 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.826605    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"]
Oct 29 22:15:21.831085 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.831064    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-fd628"]
Oct 29 22:15:21.992018 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.991989    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fbcc15a-7f6f-4742-aabb-916bb609889a" path="/var/lib/kubelet/pods/2fbcc15a-7f6f-4742-aabb-916bb609889a/volumes"
Oct 29 22:15:21.992525 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.992509    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a89ba55-981b-475e-a86c-833c547e65d5" path="/var/lib/kubelet/pods/5a89ba55-981b-475e-a86c-833c547e65d5/volumes"
Oct 29 22:15:21.992969 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:21.992956    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df402fae-12c4-4ed9-9fb1-226a51f66c18" path="/var/lib/kubelet/pods/df402fae-12c4-4ed9-9fb1-226a51f66c18/volumes"
Oct 29 22:15:22.922689 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:22.922661    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:25.012911 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:25.012878    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:25.127066 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:25.127029    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:26.707823 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:26.707783    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:27.176818 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:27.176778    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:28.335545 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:28.335505    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:30.308127 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:30.308096    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:31.904758 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:31.904725    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:32.423019 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:32.422973    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:33.738272 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:33.738242    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:34.100031 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:34.099985    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:34.400889 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:34.400802    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:35.482681 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:35.482647    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"]
Oct 29 22:15:35.483185 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:35.482864    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn" podUID="5cb930ab-251e-4279-9963-f6d0d02b5b98" containerName="istio-proxy" containerID="cri-o://17ee9f1d9e0b7ac1bb816e877090da880f55c29ce36640fc62c157f18d7d9f07" gracePeriod=30
Oct 29 22:15:35.813581 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:35.813496    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:37.342602 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:37.342515    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:37.468442 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:37.468392    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:39.146934 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:39.146900    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:40.340186 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:40.339203    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:40.474253 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:40.474219    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:40.664072 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:40.664033    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"]
Oct 29 22:15:40.664672 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:40.664620    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w" podUID="b8b9de66-29e2-4895-9ebe-b006b55614c2" containerName="istio-proxy" containerID="cri-o://fbc7c0e6a7675bfeb4e2a396b7119b3cb0dcc1968f248fc03288c53ac940a124" gracePeriod=30
Oct 29 22:15:40.677338 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:40.677312    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"]
Oct 29 22:15:40.677852 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:40.677803    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8" podUID="73cee1bc-536f-4cc5-8874-7409ae9f739c" containerName="istio-proxy" containerID="cri-o://49dadb9142d5a592356706cd637ec35879e0ee3284110a9746be095fd174d9be" gracePeriod=30
Oct 29 22:15:40.708863 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:40.708768    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:40.850984 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:40.850936    4550 generic.go:356] "Generic (PLEG): container finished" podID="5cb930ab-251e-4279-9963-f6d0d02b5b98" containerID="17ee9f1d9e0b7ac1bb816e877090da880f55c29ce36640fc62c157f18d7d9f07" exitCode=0
Oct 29 22:15:40.851210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:40.851088    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn" event={"ID":"5cb930ab-251e-4279-9963-f6d0d02b5b98","Type":"ContainerDied","Data":"17ee9f1d9e0b7ac1bb816e877090da880f55c29ce36640fc62c157f18d7d9f07"}
Oct 29 22:15:40.950842 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:40.950807    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 22:15:41.050103 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.050070    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-envoy\") pod \"5cb930ab-251e-4279-9963-f6d0d02b5b98\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") "
Oct 29 22:15:41.050264 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.050163    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-data\") pod \"5cb930ab-251e-4279-9963-f6d0d02b5b98\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") "
Oct 29 22:15:41.050264 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.050202    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb930ab-251e-4279-9963-f6d0d02b5b98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb930ab-251e-4279-9963-f6d0d02b5b98-istiod-ca-cert\") pod \"5cb930ab-251e-4279-9963-f6d0d02b5b98\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") "
Oct 29 22:15:41.050364 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.050296    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-token\" (UniqueName: \"kubernetes.io/projected/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-token\") pod \"5cb930ab-251e-4279-9963-f6d0d02b5b98\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") "
Oct 29 22:15:41.050402 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.050365    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-credential-socket\") pod \"5cb930ab-251e-4279-9963-f6d0d02b5b98\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") "
Oct 29 22:15:41.050452 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.050398    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-workload-certs\") pod \"5cb930ab-251e-4279-9963-f6d0d02b5b98\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") "
Oct 29 22:15:41.050502 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.050456    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb930ab-251e-4279-9963-f6d0d02b5b98-kube-api-access-szphr\" (UniqueName: \"kubernetes.io/projected/5cb930ab-251e-4279-9963-f6d0d02b5b98-kube-api-access-szphr\") pod \"5cb930ab-251e-4279-9963-f6d0d02b5b98\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") "
Oct 29 22:15:41.050562 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.050489    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-data" pod "5cb930ab-251e-4279-9963-f6d0d02b5b98" (UID: "5cb930ab-251e-4279-9963-f6d0d02b5b98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:41.050562 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.050503    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-workload-socket\") pod \"5cb930ab-251e-4279-9963-f6d0d02b5b98\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") "
Oct 29 22:15:41.050682 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.050579    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-podinfo\") pod \"5cb930ab-251e-4279-9963-f6d0d02b5b98\" (UID: \"5cb930ab-251e-4279-9963-f6d0d02b5b98\") "
Oct 29 22:15:41.050745 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.050676    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-credential-socket" pod "5cb930ab-251e-4279-9963-f6d0d02b5b98" (UID: "5cb930ab-251e-4279-9963-f6d0d02b5b98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:41.050745 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.050714    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb930ab-251e-4279-9963-f6d0d02b5b98-istiod-ca-cert" pod "5cb930ab-251e-4279-9963-f6d0d02b5b98" (UID: "5cb930ab-251e-4279-9963-f6d0d02b5b98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:41.050745 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.050719    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-workload-socket" pod "5cb930ab-251e-4279-9963-f6d0d02b5b98" (UID: "5cb930ab-251e-4279-9963-f6d0d02b5b98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:41.050904 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.050832    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-workload-certs" pod "5cb930ab-251e-4279-9963-f6d0d02b5b98" (UID: "5cb930ab-251e-4279-9963-f6d0d02b5b98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:41.051067 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.051049    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:41.051203 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.051069    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:41.051203 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.051078    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:41.051203 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.051086    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:41.051203 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.051098    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb930ab-251e-4279-9963-f6d0d02b5b98-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:41.052426 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.052405    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-envoy" pod "5cb930ab-251e-4279-9963-f6d0d02b5b98" (UID: "5cb930ab-251e-4279-9963-f6d0d02b5b98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:41.052577 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.052559    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-token" pod "5cb930ab-251e-4279-9963-f6d0d02b5b98" (UID: "5cb930ab-251e-4279-9963-f6d0d02b5b98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:41.052682 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.052665    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb930ab-251e-4279-9963-f6d0d02b5b98-kube-api-access-szphr" pod "5cb930ab-251e-4279-9963-f6d0d02b5b98" (UID: "5cb930ab-251e-4279-9963-f6d0d02b5b98"). InnerVolumeSpecName "kube-api-access-szphr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:41.052717 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.052677    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-podinfo" pod "5cb930ab-251e-4279-9963-f6d0d02b5b98" (UID: "5cb930ab-251e-4279-9963-f6d0d02b5b98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:41.152539 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.152506    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szphr\" (UniqueName: \"kubernetes.io/projected/5cb930ab-251e-4279-9963-f6d0d02b5b98-kube-api-access-szphr\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:41.152539 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.152531    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:41.152539 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.152543    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:41.152774 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.152551    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb930ab-251e-4279-9963-f6d0d02b5b98-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:41.491329 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:15:41.491290    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:15:41.592673 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.592617    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:41.703032 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.702992    4550 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:15:41.703207 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.703057    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w" podUID="b8b9de66-29e2-4895-9ebe-b006b55614c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:15:41.857587 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.857559    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"
Oct 29 22:15:41.857764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.857589    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn" event={"ID":"5cb930ab-251e-4279-9963-f6d0d02b5b98","Type":"ContainerDied","Data":"161358df064570e9791834ba48c7d5644f5eb1050a8e3ae21d5293a0630ca5e5"}
Oct 29 22:15:41.857764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.857659    4550 scope.go:122] "RemoveContainer" containerID="17ee9f1d9e0b7ac1bb816e877090da880f55c29ce36640fc62c157f18d7d9f07"
Oct 29 22:15:41.907988 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.907960    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"]
Oct 29 22:15:41.919869 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.919848    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svcsn"]
Oct 29 22:15:41.988555 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:41.988526    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb930ab-251e-4279-9963-f6d0d02b5b98" path="/var/lib/kubelet/pods/5cb930ab-251e-4279-9963-f6d0d02b5b98/volumes"
Oct 29 22:15:43.498982 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:43.498951    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:45.125505 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.125474    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:45.886422 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.886381    4550 generic.go:356] "Generic (PLEG): container finished" podID="b8b9de66-29e2-4895-9ebe-b006b55614c2" containerID="fbc7c0e6a7675bfeb4e2a396b7119b3cb0dcc1968f248fc03288c53ac940a124" exitCode=0
Oct 29 22:15:45.886657 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.886541    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w" event={"ID":"b8b9de66-29e2-4895-9ebe-b006b55614c2","Type":"ContainerDied","Data":"fbc7c0e6a7675bfeb4e2a396b7119b3cb0dcc1968f248fc03288c53ac940a124"}
Oct 29 22:15:45.888629 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.888600    4550 generic.go:356] "Generic (PLEG): container finished" podID="73cee1bc-536f-4cc5-8874-7409ae9f739c" containerID="49dadb9142d5a592356706cd637ec35879e0ee3284110a9746be095fd174d9be" exitCode=0
Oct 29 22:15:45.888768 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.888671    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8" event={"ID":"73cee1bc-536f-4cc5-8874-7409ae9f739c","Type":"ContainerDied","Data":"49dadb9142d5a592356706cd637ec35879e0ee3284110a9746be095fd174d9be"}
Oct 29 22:15:45.948760 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.948700    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:15:45.971893 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.971872    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:15:45.998961 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.998933    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-credential-socket\") pod \"73cee1bc-536f-4cc5-8874-7409ae9f739c\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") "
Oct 29 22:15:45.999089 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.999048    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-workload-certs\") pod \"73cee1bc-536f-4cc5-8874-7409ae9f739c\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") "
Oct 29 22:15:45.999193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.999090    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73cee1bc-536f-4cc5-8874-7409ae9f739c-kube-api-access-rvfds\" (UniqueName: \"kubernetes.io/projected/73cee1bc-536f-4cc5-8874-7409ae9f739c-kube-api-access-rvfds\") pod \"73cee1bc-536f-4cc5-8874-7409ae9f739c\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") "
Oct 29 22:15:45.999193 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.999186    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-podinfo\") pod \"73cee1bc-536f-4cc5-8874-7409ae9f739c\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") "
Oct 29 22:15:45.999328 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.999231    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-workload-socket\") pod \"73cee1bc-536f-4cc5-8874-7409ae9f739c\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") "
Oct 29 22:15:45.999328 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.999269    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73cee1bc-536f-4cc5-8874-7409ae9f739c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73cee1bc-536f-4cc5-8874-7409ae9f739c-istiod-ca-cert\") pod \"73cee1bc-536f-4cc5-8874-7409ae9f739c\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") "
Oct 29 22:15:45.999442 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.999351    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-envoy\") pod \"73cee1bc-536f-4cc5-8874-7409ae9f739c\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") "
Oct 29 22:15:45.999442 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.999380    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-token\" (UniqueName: \"kubernetes.io/projected/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-token\") pod \"73cee1bc-536f-4cc5-8874-7409ae9f739c\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") "
Oct 29 22:15:45.999442 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.999378    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-credential-socket" pod "73cee1bc-536f-4cc5-8874-7409ae9f739c" (UID: "73cee1bc-536f-4cc5-8874-7409ae9f739c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:45.999607 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.999439    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-data\") pod \"73cee1bc-536f-4cc5-8874-7409ae9f739c\" (UID: \"73cee1bc-536f-4cc5-8874-7409ae9f739c\") "
Oct 29 22:15:45.999673 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.999618    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-workload-socket" pod "73cee1bc-536f-4cc5-8874-7409ae9f739c" (UID: "73cee1bc-536f-4cc5-8874-7409ae9f739c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:45.999823 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.999805    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:45.999896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.999842    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-data" pod "73cee1bc-536f-4cc5-8874-7409ae9f739c" (UID: "73cee1bc-536f-4cc5-8874-7409ae9f739c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:45.999896 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:45.999828    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.000091 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.000072    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-workload-certs" pod "73cee1bc-536f-4cc5-8874-7409ae9f739c" (UID: "73cee1bc-536f-4cc5-8874-7409ae9f739c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.000223 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.000205    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73cee1bc-536f-4cc5-8874-7409ae9f739c-istiod-ca-cert" pod "73cee1bc-536f-4cc5-8874-7409ae9f739c" (UID: "73cee1bc-536f-4cc5-8874-7409ae9f739c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:46.002133 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.002105    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-envoy" pod "73cee1bc-536f-4cc5-8874-7409ae9f739c" (UID: "73cee1bc-536f-4cc5-8874-7409ae9f739c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.002345 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.002282    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73cee1bc-536f-4cc5-8874-7409ae9f739c-kube-api-access-rvfds" pod "73cee1bc-536f-4cc5-8874-7409ae9f739c" (UID: "73cee1bc-536f-4cc5-8874-7409ae9f739c"). InnerVolumeSpecName "kube-api-access-rvfds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.002415 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.002390    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-token" pod "73cee1bc-536f-4cc5-8874-7409ae9f739c" (UID: "73cee1bc-536f-4cc5-8874-7409ae9f739c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.002456 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.002445    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-podinfo" pod "73cee1bc-536f-4cc5-8874-7409ae9f739c" (UID: "73cee1bc-536f-4cc5-8874-7409ae9f739c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:46.100979 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.100949    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-workload-certs\") pod \"b8b9de66-29e2-4895-9ebe-b006b55614c2\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") "
Oct 29 22:15:46.101177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.100986    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-podinfo\") pod \"b8b9de66-29e2-4895-9ebe-b006b55614c2\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") "
Oct 29 22:15:46.101177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101010    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-envoy\") pod \"b8b9de66-29e2-4895-9ebe-b006b55614c2\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") "
Oct 29 22:15:46.101177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101109    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-workload-socket\") pod \"b8b9de66-29e2-4895-9ebe-b006b55614c2\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") "
Oct 29 22:15:46.101177 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101164    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b9de66-29e2-4895-9ebe-b006b55614c2-kube-api-access-g9m4d\" (UniqueName: \"kubernetes.io/projected/b8b9de66-29e2-4895-9ebe-b006b55614c2-kube-api-access-g9m4d\") pod \"b8b9de66-29e2-4895-9ebe-b006b55614c2\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") "
Oct 29 22:15:46.101420 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101231    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b9de66-29e2-4895-9ebe-b006b55614c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b9de66-29e2-4895-9ebe-b006b55614c2-istiod-ca-cert\") pod \"b8b9de66-29e2-4895-9ebe-b006b55614c2\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") "
Oct 29 22:15:46.101420 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101267    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-token\" (UniqueName: \"kubernetes.io/projected/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-token\") pod \"b8b9de66-29e2-4895-9ebe-b006b55614c2\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") "
Oct 29 22:15:46.101420 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101314    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-credential-socket\") pod \"b8b9de66-29e2-4895-9ebe-b006b55614c2\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") "
Oct 29 22:15:46.101420 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101328    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-workload-certs" pod "b8b9de66-29e2-4895-9ebe-b006b55614c2" (UID: "b8b9de66-29e2-4895-9ebe-b006b55614c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.101420 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101349    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-data\") pod \"b8b9de66-29e2-4895-9ebe-b006b55614c2\" (UID: \"b8b9de66-29e2-4895-9ebe-b006b55614c2\") "
Oct 29 22:15:46.101699 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101439    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-workload-socket" pod "b8b9de66-29e2-4895-9ebe-b006b55614c2" (UID: "b8b9de66-29e2-4895-9ebe-b006b55614c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.101699 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101680    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-credential-socket" pod "b8b9de66-29e2-4895-9ebe-b006b55614c2" (UID: "b8b9de66-29e2-4895-9ebe-b006b55614c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.101812 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101700    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b9de66-29e2-4895-9ebe-b006b55614c2-istiod-ca-cert" pod "b8b9de66-29e2-4895-9ebe-b006b55614c2" (UID: "b8b9de66-29e2-4895-9ebe-b006b55614c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:46.101870 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101815    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b9de66-29e2-4895-9ebe-b006b55614c2-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.101870 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101864    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.101987 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101880    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.101987 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101881    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-data" pod "b8b9de66-29e2-4895-9ebe-b006b55614c2" (UID: "b8b9de66-29e2-4895-9ebe-b006b55614c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.101987 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101893    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.101987 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101911    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.101987 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101925    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.101987 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101938    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73cee1bc-536f-4cc5-8874-7409ae9f739c-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.101987 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101954    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvfds\" (UniqueName: \"kubernetes.io/projected/73cee1bc-536f-4cc5-8874-7409ae9f739c-kube-api-access-rvfds\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.101987 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101968    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73cee1bc-536f-4cc5-8874-7409ae9f739c-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.101987 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101984    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.102390 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.101996    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73cee1bc-536f-4cc5-8874-7409ae9f739c-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.103760 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.103724    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-envoy" pod "b8b9de66-29e2-4895-9ebe-b006b55614c2" (UID: "b8b9de66-29e2-4895-9ebe-b006b55614c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.103861 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.103811    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-podinfo" pod "b8b9de66-29e2-4895-9ebe-b006b55614c2" (UID: "b8b9de66-29e2-4895-9ebe-b006b55614c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:46.103936 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.103920    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-token" pod "b8b9de66-29e2-4895-9ebe-b006b55614c2" (UID: "b8b9de66-29e2-4895-9ebe-b006b55614c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.104182 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.104163    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b9de66-29e2-4895-9ebe-b006b55614c2-kube-api-access-g9m4d" pod "b8b9de66-29e2-4895-9ebe-b006b55614c2" (UID: "b8b9de66-29e2-4895-9ebe-b006b55614c2"). InnerVolumeSpecName "kube-api-access-g9m4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.203489 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.203430    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9m4d\" (UniqueName: \"kubernetes.io/projected/b8b9de66-29e2-4895-9ebe-b006b55614c2-kube-api-access-g9m4d\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.203489 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.203453    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.203489 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.203463    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.203489 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.203471    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.203489 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.203479    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b9de66-29e2-4895-9ebe-b006b55614c2-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.894958 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.894926    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"
Oct 29 22:15:46.895189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.894958    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w" event={"ID":"b8b9de66-29e2-4895-9ebe-b006b55614c2","Type":"ContainerDied","Data":"faec6456ca4b6a39fbc833f56f0cbec989b31df4114bb5306aa610510f3a0de3"}
Oct 29 22:15:46.895189 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.895029    4550 scope.go:122] "RemoveContainer" containerID="fbc7c0e6a7675bfeb4e2a396b7119b3cb0dcc1968f248fc03288c53ac940a124"
Oct 29 22:15:46.896486 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.896255    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8" event={"ID":"73cee1bc-536f-4cc5-8874-7409ae9f739c","Type":"ContainerDied","Data":"14653c3763b4a6583a5b878643b69a432a3bad9b70287f2915a97b2b162395eb"}
Oct 29 22:15:46.896486 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.896297    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"
Oct 29 22:15:46.904431 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.904351    4550 scope.go:122] "RemoveContainer" containerID="49dadb9142d5a592356706cd637ec35879e0ee3284110a9746be095fd174d9be"
Oct 29 22:15:46.957427 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.957373    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"]
Oct 29 22:15:46.969487 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.969461    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-v7cn8"]
Oct 29 22:15:46.995242 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.995209    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"]
Oct 29 22:15:46.996968 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:46.996944    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-45z2w"]
Oct 29 22:15:47.115001 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:47.114956    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:47.987586 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:47.987554    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73cee1bc-536f-4cc5-8874-7409ae9f739c" path="/var/lib/kubelet/pods/73cee1bc-536f-4cc5-8874-7409ae9f739c/volumes"
Oct 29 22:15:47.988018 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:47.988003    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b9de66-29e2-4895-9ebe-b006b55614c2" path="/var/lib/kubelet/pods/b8b9de66-29e2-4895-9ebe-b006b55614c2/volumes"
Oct 29 22:15:48.701624 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:48.701579    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:50.332716 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:50.332680    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:51.411844 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:51.411811    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:52.315022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:52.314992    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:53.949133 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:53.949086    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:55.338518 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:55.338486    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:55.514262 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:55.514229    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:55.635096 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:55.635011    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:55.833348 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:55.833306    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:55.955887 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:55.955805    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:56.068854 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:56.068815    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:56.462108 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:56.462075    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:57.619827 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:57.619790    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:59.112561 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:15:59.112530    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:00.712113 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:00.712081    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:02.371166 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:02.368684    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:03.257806 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:03.257769    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:03.389967 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:03.389931    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:03.598074 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:03.598039    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"]
Oct 29 22:16:03.598633 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:03.598588    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm" podUID="8f0d50a6-d0cf-44ae-86e2-32558679df0b" containerName="istio-proxy" containerID="cri-o://8b4497459c15992a3e0bc3cb6deb10c58c2b682e64971a770d8c0d1a00b50551" gracePeriod=30
Oct 29 22:16:03.639791 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:03.639749    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:04.101105 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:04.101075    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:04.218044 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:04.217987    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:05.824375 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:05.824337    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:07.362750 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:07.362714    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:16:07.363221 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:07.362774    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm" podUID="8f0d50a6-d0cf-44ae-86e2-32558679df0b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:16:07.504263 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:07.504216    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:09.077180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.068848    4550 generic.go:356] "Generic (PLEG): container finished" podID="8f0d50a6-d0cf-44ae-86e2-32558679df0b" containerID="8b4497459c15992a3e0bc3cb6deb10c58c2b682e64971a770d8c0d1a00b50551" exitCode=0
Oct 29 22:16:09.077180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.069034    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm" event={"ID":"8f0d50a6-d0cf-44ae-86e2-32558679df0b","Type":"ContainerDied","Data":"8b4497459c15992a3e0bc3cb6deb10c58c2b682e64971a770d8c0d1a00b50551"}
Oct 29 22:16:09.077180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.069096    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm" event={"ID":"8f0d50a6-d0cf-44ae-86e2-32558679df0b","Type":"ContainerDied","Data":"6717d3d4a1073ee7ec481d141accdef46f4807db448dad80270ee7883cb16546"}
Oct 29 22:16:09.077180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.069124    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6717d3d4a1073ee7ec481d141accdef46f4807db448dad80270ee7883cb16546"
Oct 29 22:16:09.077888 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.077498    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:16:09.135824 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.131132    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"]
Oct 29 22:16:09.234231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.233954    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-data\") pod \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") "
Oct 29 22:16:09.234231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.234016    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-podinfo\") pod \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") "
Oct 29 22:16:09.234231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.234108    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-workload-socket\") pod \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") "
Oct 29 22:16:09.234231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.234157    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-envoy\") pod \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") "
Oct 29 22:16:09.234231 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.234204    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-workload-certs\") pod \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") "
Oct 29 22:16:09.234624 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.234248    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0d50a6-d0cf-44ae-86e2-32558679df0b-kube-api-access-pdjdp\" (UniqueName: \"kubernetes.io/projected/8f0d50a6-d0cf-44ae-86e2-32558679df0b-kube-api-access-pdjdp\") pod \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") "
Oct 29 22:16:09.234624 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.234326    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istiod-ca-cert\") pod \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") "
Oct 29 22:16:09.234624 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.234355    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-credential-socket\") pod \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") "
Oct 29 22:16:09.234624 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.234380    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-token\" (UniqueName: \"kubernetes.io/projected/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-token\") pod \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\" (UID: \"8f0d50a6-d0cf-44ae-86e2-32558679df0b\") "
Oct 29 22:16:09.240349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.238900    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istiod-ca-cert" pod "8f0d50a6-d0cf-44ae-86e2-32558679df0b" (UID: "8f0d50a6-d0cf-44ae-86e2-32558679df0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:09.240349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.239264    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-credential-socket" pod "8f0d50a6-d0cf-44ae-86e2-32558679df0b" (UID: "8f0d50a6-d0cf-44ae-86e2-32558679df0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.240349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.239806    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-workload-certs" pod "8f0d50a6-d0cf-44ae-86e2-32558679df0b" (UID: "8f0d50a6-d0cf-44ae-86e2-32558679df0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.240349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.239830    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-workload-socket" pod "8f0d50a6-d0cf-44ae-86e2-32558679df0b" (UID: "8f0d50a6-d0cf-44ae-86e2-32558679df0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.240349 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.240266    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:09.242959 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.242908    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-podinfo" pod "8f0d50a6-d0cf-44ae-86e2-32558679df0b" (UID: "8f0d50a6-d0cf-44ae-86e2-32558679df0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:09.242959 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.242925    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-data" pod "8f0d50a6-d0cf-44ae-86e2-32558679df0b" (UID: "8f0d50a6-d0cf-44ae-86e2-32558679df0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.248766 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.247589    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0d50a6-d0cf-44ae-86e2-32558679df0b-kube-api-access-pdjdp" pod "8f0d50a6-d0cf-44ae-86e2-32558679df0b" (UID: "8f0d50a6-d0cf-44ae-86e2-32558679df0b"). InnerVolumeSpecName "kube-api-access-pdjdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:09.252591 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.252550    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-token" pod "8f0d50a6-d0cf-44ae-86e2-32558679df0b" (UID: "8f0d50a6-d0cf-44ae-86e2-32558679df0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:09.253321 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.253270    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-envoy" pod "8f0d50a6-d0cf-44ae-86e2-32558679df0b" (UID: "8f0d50a6-d0cf-44ae-86e2-32558679df0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.337642 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.337604    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.337642 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.337639    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdjdp\" (UniqueName: \"kubernetes.io/projected/8f0d50a6-d0cf-44ae-86e2-32558679df0b-kube-api-access-pdjdp\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.337898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.337653    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.337898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.337665    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.337898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.337677    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.337898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.337688    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.337898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.337698    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.337898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.337709    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.337898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.337719    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f0d50a6-d0cf-44ae-86e2-32558679df0b-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.379164 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:09.375034    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:10.073412 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:10.073379    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"
Oct 29 22:16:10.135845 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:10.135808    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"]
Oct 29 22:16:10.139834 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:10.139809    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-d42vm"]
Oct 29 22:16:11.008968 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:11.008933    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:11.490313 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:16:11.490275    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:16:11.987006 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:11.986971    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0d50a6-d0cf-44ae-86e2-32558679df0b" path="/var/lib/kubelet/pods/8f0d50a6-d0cf-44ae-86e2-32558679df0b/volumes"
Oct 29 22:16:12.518330 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:12.518298    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:14.023567 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:14.023512    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:14.548044 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:14.548009    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:14.904237 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:14.904192    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:15.021163 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:15.021083    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:15.782568 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:15.782534    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:16.604082 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:16.604052    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:16.718973 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:16.718943    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:24.278742 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:24.278700    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" containerID="cri-o://746dc9186f20b1de1da56f58ea4219aed87a1c0d2e7cf50914e2e548e36f8751" gracePeriod=45
Oct 29 22:16:24.418171 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:24.417583    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 29 22:16:24.881638 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:24.881330    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:16:24.939267 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:24.939164    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-dshm\") pod \"eff86f8c-7f09-43cb-bffc-889eb6e56181\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") "
Oct 29 22:16:24.939438 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:24.939263    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-home\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-home\") pod \"eff86f8c-7f09-43cb-bffc-889eb6e56181\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") "
Oct 29 22:16:24.939438 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:24.939343    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-model-cache\") pod \"eff86f8c-7f09-43cb-bffc-889eb6e56181\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") "
Oct 29 22:16:24.939438 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:24.939378    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-tmp-dir\") pod \"eff86f8c-7f09-43cb-bffc-889eb6e56181\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") "
Oct 29 22:16:24.939438 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:24.939422    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff86f8c-7f09-43cb-bffc-889eb6e56181-kube-api-access-lmdk2\" (UniqueName: \"kubernetes.io/projected/eff86f8c-7f09-43cb-bffc-889eb6e56181-kube-api-access-lmdk2\") pod \"eff86f8c-7f09-43cb-bffc-889eb6e56181\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") "
Oct 29 22:16:24.939694 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:24.939472    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eff86f8c-7f09-43cb-bffc-889eb6e56181-tls-certs\" (UniqueName: \"kubernetes.io/secret/eff86f8c-7f09-43cb-bffc-889eb6e56181-tls-certs\") pod \"eff86f8c-7f09-43cb-bffc-889eb6e56181\" (UID: \"eff86f8c-7f09-43cb-bffc-889eb6e56181\") "
Oct 29 22:16:24.940039 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:24.940014    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-model-cache" pod "eff86f8c-7f09-43cb-bffc-889eb6e56181" (UID: "eff86f8c-7f09-43cb-bffc-889eb6e56181"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:24.940923 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:24.940509    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-tmp-dir" pod "eff86f8c-7f09-43cb-bffc-889eb6e56181" (UID: "eff86f8c-7f09-43cb-bffc-889eb6e56181"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:24.940923 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:24.940864    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-home" pod "eff86f8c-7f09-43cb-bffc-889eb6e56181" (UID: "eff86f8c-7f09-43cb-bffc-889eb6e56181"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:24.948342 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:24.947642    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff86f8c-7f09-43cb-bffc-889eb6e56181-tls-certs" pod "eff86f8c-7f09-43cb-bffc-889eb6e56181" (UID: "eff86f8c-7f09-43cb-bffc-889eb6e56181"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:16:24.948342 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:24.948296    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-dshm" pod "eff86f8c-7f09-43cb-bffc-889eb6e56181" (UID: "eff86f8c-7f09-43cb-bffc-889eb6e56181"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:24.948664 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:24.948623    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff86f8c-7f09-43cb-bffc-889eb6e56181-kube-api-access-lmdk2" pod "eff86f8c-7f09-43cb-bffc-889eb6e56181" (UID: "eff86f8c-7f09-43cb-bffc-889eb6e56181"). InnerVolumeSpecName "kube-api-access-lmdk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:25.041397 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.041364    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-model-cache\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.041397 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.041394    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-tmp-dir\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.041611 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.041408    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmdk2\" (UniqueName: \"kubernetes.io/projected/eff86f8c-7f09-43cb-bffc-889eb6e56181-kube-api-access-lmdk2\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.041611 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.041423    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff86f8c-7f09-43cb-bffc-889eb6e56181-tls-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.041611 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.041437    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-dshm\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.041611 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.041447    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff86f8c-7f09-43cb-bffc-889eb6e56181-home\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.154246 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.154212    4550 generic.go:356] "Generic (PLEG): container finished" podID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerID="746dc9186f20b1de1da56f58ea4219aed87a1c0d2e7cf50914e2e548e36f8751" exitCode=0
Oct 29 22:16:25.154412 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.154259    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"
Oct 29 22:16:25.154412 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.154299    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" event={"ID":"eff86f8c-7f09-43cb-bffc-889eb6e56181","Type":"ContainerDied","Data":"746dc9186f20b1de1da56f58ea4219aed87a1c0d2e7cf50914e2e548e36f8751"}
Oct 29 22:16:25.154412 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.154345    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7" event={"ID":"eff86f8c-7f09-43cb-bffc-889eb6e56181","Type":"ContainerDied","Data":"20b586f6057b1b7f7d62123d13bba7b56ddc497372aca004507ac8bda20fcff3"}
Oct 29 22:16:25.154412 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.154368    4550 scope.go:122] "RemoveContainer" containerID="746dc9186f20b1de1da56f58ea4219aed87a1c0d2e7cf50914e2e548e36f8751"
Oct 29 22:16:25.163447 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.163428    4550 scope.go:122] "RemoveContainer" containerID="746dc9186f20b1de1da56f58ea4219aed87a1c0d2e7cf50914e2e548e36f8751"
Oct 29 22:16:25.163707 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:16:25.163687    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746dc9186f20b1de1da56f58ea4219aed87a1c0d2e7cf50914e2e548e36f8751\": container with ID starting with 746dc9186f20b1de1da56f58ea4219aed87a1c0d2e7cf50914e2e548e36f8751 not found: ID does not exist" containerID="746dc9186f20b1de1da56f58ea4219aed87a1c0d2e7cf50914e2e548e36f8751"
Oct 29 22:16:25.163764 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.163714    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746dc9186f20b1de1da56f58ea4219aed87a1c0d2e7cf50914e2e548e36f8751"} err="failed to get container status \"746dc9186f20b1de1da56f58ea4219aed87a1c0d2e7cf50914e2e548e36f8751\": rpc error: code = NotFound desc = could not find container \"746dc9186f20b1de1da56f58ea4219aed87a1c0d2e7cf50914e2e548e36f8751\": container with ID starting with 746dc9186f20b1de1da56f58ea4219aed87a1c0d2e7cf50914e2e548e36f8751 not found: ID does not exist"
Oct 29 22:16:25.207103 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.207046    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"]
Oct 29 22:16:25.219083 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.219060    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/premium-w3-8a9e5f-kserve-75dffd5c8f-mkrl7"]
Oct 29 22:16:25.987946 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:25.987913    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" path="/var/lib/kubelet/pods/eff86f8c-7f09-43cb-bffc-889eb6e56181/volumes"
Oct 29 22:16:32.318539 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.318504    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"]
Oct 29 22:16:32.319077 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.318740    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5" podUID="8cf254b1-d374-4b55-8fec-d04f6b6501c1" containerName="maas-api" containerID="cri-o://09090d2c0196811d1d5c4e28d40ca3cb4980753df4964c5a25d4a1b98216f97b" gracePeriod=30
Oct 29 22:16:32.417337 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.417304    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:32.615746 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.615717    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 22:16:32.733962 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.733920    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf254b1-d374-4b55-8fec-d04f6b6501c1-kube-api-access-528gl\" (UniqueName: \"kubernetes.io/projected/8cf254b1-d374-4b55-8fec-d04f6b6501c1-kube-api-access-528gl\") pod \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") "
Oct 29 22:16:32.734210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.733995    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cf254b1-d374-4b55-8fec-d04f6b6501c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf254b1-d374-4b55-8fec-d04f6b6501c1-metrics-certs\") pod \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") "
Oct 29 22:16:32.734210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.734073    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cf254b1-d374-4b55-8fec-d04f6b6501c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf254b1-d374-4b55-8fec-d04f6b6501c1-maas-api-tls\") pod \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") "
Oct 29 22:16:32.734210 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.734182    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf254b1-d374-4b55-8fec-d04f6b6501c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf254b1-d374-4b55-8fec-d04f6b6501c1-openshift-service-ca\") pod \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\" (UID: \"8cf254b1-d374-4b55-8fec-d04f6b6501c1\") "
Oct 29 22:16:32.736864 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.736829    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf254b1-d374-4b55-8fec-d04f6b6501c1-openshift-service-ca" pod "8cf254b1-d374-4b55-8fec-d04f6b6501c1" (UID: "8cf254b1-d374-4b55-8fec-d04f6b6501c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:32.742674 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.742632    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf254b1-d374-4b55-8fec-d04f6b6501c1-maas-api-tls" pod "8cf254b1-d374-4b55-8fec-d04f6b6501c1" (UID: "8cf254b1-d374-4b55-8fec-d04f6b6501c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:16:32.743476 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.743439    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf254b1-d374-4b55-8fec-d04f6b6501c1-metrics-certs" pod "8cf254b1-d374-4b55-8fec-d04f6b6501c1" (UID: "8cf254b1-d374-4b55-8fec-d04f6b6501c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:16:32.743726 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.743697    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf254b1-d374-4b55-8fec-d04f6b6501c1-kube-api-access-528gl" pod "8cf254b1-d374-4b55-8fec-d04f6b6501c1" (UID: "8cf254b1-d374-4b55-8fec-d04f6b6501c1"). InnerVolumeSpecName "kube-api-access-528gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:32.835791 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.835752    4550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf254b1-d374-4b55-8fec-d04f6b6501c1-openshift-service-ca\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:32.835791 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.835793    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-528gl\" (UniqueName: \"kubernetes.io/projected/8cf254b1-d374-4b55-8fec-d04f6b6501c1-kube-api-access-528gl\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:32.836001 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.835808    4550 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf254b1-d374-4b55-8fec-d04f6b6501c1-metrics-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:32.836001 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.835824    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf254b1-d374-4b55-8fec-d04f6b6501c1-maas-api-tls\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:32.908872 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:32.908792    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:33.031062 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:33.031027    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:33.215497 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:33.215406    4550 generic.go:356] "Generic (PLEG): container finished" podID="8cf254b1-d374-4b55-8fec-d04f6b6501c1" containerID="09090d2c0196811d1d5c4e28d40ca3cb4980753df4964c5a25d4a1b98216f97b" exitCode=0
Oct 29 22:16:33.215497 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:33.215468    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5" event={"ID":"8cf254b1-d374-4b55-8fec-d04f6b6501c1","Type":"ContainerDied","Data":"09090d2c0196811d1d5c4e28d40ca3cb4980753df4964c5a25d4a1b98216f97b"}
Oct 29 22:16:33.215713 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:33.215505    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"
Oct 29 22:16:33.215713 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:33.215647    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5" event={"ID":"8cf254b1-d374-4b55-8fec-d04f6b6501c1","Type":"ContainerDied","Data":"d37c2c8d601b073a276f963cf41634921c515e3811f7210df6dccb80722fae03"}
Oct 29 22:16:33.215713 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:33.215670    4550 scope.go:122] "RemoveContainer" containerID="09090d2c0196811d1d5c4e28d40ca3cb4980753df4964c5a25d4a1b98216f97b"
Oct 29 22:16:33.225099 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:33.225078    4550 scope.go:122] "RemoveContainer" containerID="09090d2c0196811d1d5c4e28d40ca3cb4980753df4964c5a25d4a1b98216f97b"
Oct 29 22:16:33.225413 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:16:33.225394    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09090d2c0196811d1d5c4e28d40ca3cb4980753df4964c5a25d4a1b98216f97b\": container with ID starting with 09090d2c0196811d1d5c4e28d40ca3cb4980753df4964c5a25d4a1b98216f97b not found: ID does not exist" containerID="09090d2c0196811d1d5c4e28d40ca3cb4980753df4964c5a25d4a1b98216f97b"
Oct 29 22:16:33.225473 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:33.225423    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09090d2c0196811d1d5c4e28d40ca3cb4980753df4964c5a25d4a1b98216f97b"} err="failed to get container status \"09090d2c0196811d1d5c4e28d40ca3cb4980753df4964c5a25d4a1b98216f97b\": rpc error: code = NotFound desc = could not find container \"09090d2c0196811d1d5c4e28d40ca3cb4980753df4964c5a25d4a1b98216f97b\": container with ID starting with 09090d2c0196811d1d5c4e28d40ca3cb4980753df4964c5a25d4a1b98216f97b not found: ID does not exist"
Oct 29 22:16:33.268267 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:33.268241    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"]
Oct 29 22:16:33.270884 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:33.270862    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8a9e5f-5dfd56d8-rwwt5"]
Oct 29 22:16:33.539054 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:33.538977    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:33.988205 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:33.988171    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf254b1-d374-4b55-8fec-d04f6b6501c1" path="/var/lib/kubelet/pods/8cf254b1-d374-4b55-8fec-d04f6b6501c1/volumes"
Oct 29 22:16:39.795872 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:39.795840    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:39.913653 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:39.913625    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:40.124191 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:40.123919    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"]
Oct 29 22:16:40.124623 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:40.124589    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerName="istio-proxy" containerID="cri-o://73823e5c9354fa4d74bf0c904d8de9ebf5d3932aeb8063e28f5f1fb38703bcfb" gracePeriod=30
Oct 29 22:16:40.138811 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:40.138774    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"]
Oct 29 22:16:40.138990 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:40.138857    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"]
Oct 29 22:16:40.139075 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:40.139051    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" podUID="a26797f1-f34c-44e3-86e9-dd0ec24af003" containerName="istio-proxy" containerID="cri-o://3d9d4df4d210aa83747257be2ea6467fcee5bd0b8644e657a3e9845aa18a4f1f" gracePeriod=30
Oct 29 22:16:40.139358 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:40.139331    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerName="istio-proxy" containerID="cri-o://bc9aa7797542acc8e82b2e5a187d3224b80f8b23c8c3cda73743ff5e9d938841" gracePeriod=30
Oct 29 22:16:40.154885 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:40.154854    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:41.489988 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:16:41.489955    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:16:41.540762 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:41.540734    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-779bff956b-4cdnx_3a9b6803-3648-4ea1-95b4-61a8b1245a7b/manager/0.log"
Oct 29 22:16:41.983626 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:41.983596    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-57f5c66455-wqzgt_09c103da-69b2-47c3-a35e-2dc18f6efee7/manager/0.log"
Oct 29 22:16:42.958278 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:42.958242    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:43.132339 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:43.132306    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:43.262480 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:43.262390    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:43.384079 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:43.384050    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:43.444257 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:43.444222    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_mcp-gateway-controller-6c98b995c5-kczzc_c6436350-e6cf-4ef9-a0ac-2cc72b34fb0b/mcp-controller/0.log"
Oct 29 22:16:44.200597 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:44.200567    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:44.609035 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:44.609007    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-f5cm2_20102b85-6195-42e4-94ea-873d5af3ad35/manager/0.log"
Oct 29 22:16:45.043953 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.043884    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-q94rt_cd3a6ac7-2936-45af-9951-a1002cbeab78/manager/0.log"
Oct 29 22:16:45.276294 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.276097    4550 generic.go:356] "Generic (PLEG): container finished" podID="a26797f1-f34c-44e3-86e9-dd0ec24af003" containerID="3d9d4df4d210aa83747257be2ea6467fcee5bd0b8644e657a3e9845aa18a4f1f" exitCode=0
Oct 29 22:16:45.276294 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.276209    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" event={"ID":"a26797f1-f34c-44e3-86e9-dd0ec24af003","Type":"ContainerDied","Data":"3d9d4df4d210aa83747257be2ea6467fcee5bd0b8644e657a3e9845aa18a4f1f"}
Oct 29 22:16:45.279135 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.279104    4550 generic.go:356] "Generic (PLEG): container finished" podID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerID="73823e5c9354fa4d74bf0c904d8de9ebf5d3932aeb8063e28f5f1fb38703bcfb" exitCode=0
Oct 29 22:16:45.279330 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.279307    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" event={"ID":"bdcde015-efb5-4689-84f3-6e65620b8a53","Type":"ContainerDied","Data":"73823e5c9354fa4d74bf0c904d8de9ebf5d3932aeb8063e28f5f1fb38703bcfb"}
Oct 29 22:16:45.281480 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.281310    4550 generic.go:356] "Generic (PLEG): container finished" podID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerID="bc9aa7797542acc8e82b2e5a187d3224b80f8b23c8c3cda73743ff5e9d938841" exitCode=0
Oct 29 22:16:45.281480 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.281382    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" event={"ID":"6ae4f671-bd04-4b77-9e76-17be23e5f5ee","Type":"ContainerDied","Data":"bc9aa7797542acc8e82b2e5a187d3224b80f8b23c8c3cda73743ff5e9d938841"}
Oct 29 22:16:45.430828 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.430803    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f6b55dfc4-hv5c5_6a47b6ed-0ade-4b53-ab33-43e52c37ed92/manager/0.log"
Oct 29 22:16:45.431535 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.431516    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 22:16:45.457235 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.457215    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 22:16:45.460685 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.460662    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 22:16:45.560741 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.560711    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-kube-api-access-69fdc\" (UniqueName: \"kubernetes.io/projected/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-kube-api-access-69fdc\") pod \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") "
Oct 29 22:16:45.560946 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.560774    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a26797f1-f34c-44e3-86e9-dd0ec24af003-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26797f1-f34c-44e3-86e9-dd0ec24af003-istiod-ca-cert\") pod \"a26797f1-f34c-44e3-86e9-dd0ec24af003\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") "
Oct 29 22:16:45.560946 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.560804    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-data\") pod \"a26797f1-f34c-44e3-86e9-dd0ec24af003\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") "
Oct 29 22:16:45.561075 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561056    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-data" pod "a26797f1-f34c-44e3-86e9-dd0ec24af003" (UID: "a26797f1-f34c-44e3-86e9-dd0ec24af003"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.561240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561123    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-podinfo\") pod \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") "
Oct 29 22:16:45.561240 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561201    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-token\" (UniqueName: \"kubernetes.io/projected/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-token\") pod \"a26797f1-f34c-44e3-86e9-dd0ec24af003\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") "
Oct 29 22:16:45.561405 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561256    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-token\" (UniqueName: \"kubernetes.io/projected/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-token\") pod \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") "
Oct 29 22:16:45.561405 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561263    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26797f1-f34c-44e3-86e9-dd0ec24af003-istiod-ca-cert" pod "a26797f1-f34c-44e3-86e9-dd0ec24af003" (UID: "a26797f1-f34c-44e3-86e9-dd0ec24af003"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:45.561405 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561287    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-envoy\") pod \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") "
Oct 29 22:16:45.561405 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561334    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-workload-certs\") pod \"bdcde015-efb5-4689-84f3-6e65620b8a53\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") "
Oct 29 22:16:45.561405 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561377    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-podinfo\") pod \"a26797f1-f34c-44e3-86e9-dd0ec24af003\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") "
Oct 29 22:16:45.561405 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561402    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26797f1-f34c-44e3-86e9-dd0ec24af003-kube-api-access-v9rvp\" (UniqueName: \"kubernetes.io/projected/a26797f1-f34c-44e3-86e9-dd0ec24af003-kube-api-access-v9rvp\") pod \"a26797f1-f34c-44e3-86e9-dd0ec24af003\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") "
Oct 29 22:16:45.561731 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561446    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-credential-socket\") pod \"a26797f1-f34c-44e3-86e9-dd0ec24af003\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") "
Oct 29 22:16:45.561731 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561486    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-envoy\") pod \"a26797f1-f34c-44e3-86e9-dd0ec24af003\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") "
Oct 29 22:16:45.561731 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561522    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-podinfo\") pod \"bdcde015-efb5-4689-84f3-6e65620b8a53\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") "
Oct 29 22:16:45.561731 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561558    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdcde015-efb5-4689-84f3-6e65620b8a53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdcde015-efb5-4689-84f3-6e65620b8a53-istiod-ca-cert\") pod \"bdcde015-efb5-4689-84f3-6e65620b8a53\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") "
Oct 29 22:16:45.561731 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561589    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-workload-certs\") pod \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") "
Oct 29 22:16:45.561731 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561662    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istiod-ca-cert\") pod \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") "
Oct 29 22:16:45.561731 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561703    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-envoy\") pod \"bdcde015-efb5-4689-84f3-6e65620b8a53\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") "
Oct 29 22:16:45.562129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561739    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-credential-socket\") pod \"bdcde015-efb5-4689-84f3-6e65620b8a53\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") "
Oct 29 22:16:45.562129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561776    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-workload-certs\") pod \"a26797f1-f34c-44e3-86e9-dd0ec24af003\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") "
Oct 29 22:16:45.562129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561815    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-credential-socket\") pod \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") "
Oct 29 22:16:45.562129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561863    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-token\" (UniqueName: \"kubernetes.io/projected/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-token\") pod \"bdcde015-efb5-4689-84f3-6e65620b8a53\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") "
Oct 29 22:16:45.562129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561895    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-workload-socket\") pod \"bdcde015-efb5-4689-84f3-6e65620b8a53\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") "
Oct 29 22:16:45.562129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561940    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-data\") pod \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") "
Oct 29 22:16:45.562129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.561980    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-workload-socket\") pod \"a26797f1-f34c-44e3-86e9-dd0ec24af003\" (UID: \"a26797f1-f34c-44e3-86e9-dd0ec24af003\") "
Oct 29 22:16:45.562129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.562026    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-data\") pod \"bdcde015-efb5-4689-84f3-6e65620b8a53\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") "
Oct 29 22:16:45.562129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.562060    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdcde015-efb5-4689-84f3-6e65620b8a53-kube-api-access-kqttt\" (UniqueName: \"kubernetes.io/projected/bdcde015-efb5-4689-84f3-6e65620b8a53-kube-api-access-kqttt\") pod \"bdcde015-efb5-4689-84f3-6e65620b8a53\" (UID: \"bdcde015-efb5-4689-84f3-6e65620b8a53\") "
Oct 29 22:16:45.562129 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.562104    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-workload-socket\") pod \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\" (UID: \"6ae4f671-bd04-4b77-9e76-17be23e5f5ee\") "
Oct 29 22:16:45.562897 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.562872    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istiod-ca-cert" pod "6ae4f671-bd04-4b77-9e76-17be23e5f5ee" (UID: "6ae4f671-bd04-4b77-9e76-17be23e5f5ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:45.563601 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.563287    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-workload-socket" pod "a26797f1-f34c-44e3-86e9-dd0ec24af003" (UID: "a26797f1-f34c-44e3-86e9-dd0ec24af003"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.563601 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.563509    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-data" pod "6ae4f671-bd04-4b77-9e76-17be23e5f5ee" (UID: "6ae4f671-bd04-4b77-9e76-17be23e5f5ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.563601 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.563530    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-credential-socket" pod "bdcde015-efb5-4689-84f3-6e65620b8a53" (UID: "bdcde015-efb5-4689-84f3-6e65620b8a53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.563863 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.563635    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-workload-certs" pod "6ae4f671-bd04-4b77-9e76-17be23e5f5ee" (UID: "6ae4f671-bd04-4b77-9e76-17be23e5f5ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.564970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.563988    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26797f1-f34c-44e3-86e9-dd0ec24af003-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.564970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.564017    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.564970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.564035    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.564970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.564049    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.564970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.564062    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.564970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.564076    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.564970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.564089    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.564970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.564195    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-workload-certs" pod "a26797f1-f34c-44e3-86e9-dd0ec24af003" (UID: "a26797f1-f34c-44e3-86e9-dd0ec24af003"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.564970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.564203    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-workload-certs" pod "bdcde015-efb5-4689-84f3-6e65620b8a53" (UID: "bdcde015-efb5-4689-84f3-6e65620b8a53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.564970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.564245    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-workload-socket" pod "bdcde015-efb5-4689-84f3-6e65620b8a53" (UID: "bdcde015-efb5-4689-84f3-6e65620b8a53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.564970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.564320    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-workload-socket" pod "6ae4f671-bd04-4b77-9e76-17be23e5f5ee" (UID: "6ae4f671-bd04-4b77-9e76-17be23e5f5ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.564970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.564407    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-credential-socket" pod "6ae4f671-bd04-4b77-9e76-17be23e5f5ee" (UID: "6ae4f671-bd04-4b77-9e76-17be23e5f5ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.564970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.564440    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdcde015-efb5-4689-84f3-6e65620b8a53-istiod-ca-cert" pod "bdcde015-efb5-4689-84f3-6e65620b8a53" (UID: "bdcde015-efb5-4689-84f3-6e65620b8a53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:45.564970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.564538    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-data" pod "bdcde015-efb5-4689-84f3-6e65620b8a53" (UID: "bdcde015-efb5-4689-84f3-6e65620b8a53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.564970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.564760    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-credential-socket" pod "a26797f1-f34c-44e3-86e9-dd0ec24af003" (UID: "a26797f1-f34c-44e3-86e9-dd0ec24af003"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.565871 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.564986    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-podinfo" pod "6ae4f671-bd04-4b77-9e76-17be23e5f5ee" (UID: "6ae4f671-bd04-4b77-9e76-17be23e5f5ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:45.565871 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.565310    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-kube-api-access-69fdc" pod "6ae4f671-bd04-4b77-9e76-17be23e5f5ee" (UID: "6ae4f671-bd04-4b77-9e76-17be23e5f5ee"). InnerVolumeSpecName "kube-api-access-69fdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.566387 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.566350    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-token" pod "a26797f1-f34c-44e3-86e9-dd0ec24af003" (UID: "a26797f1-f34c-44e3-86e9-dd0ec24af003"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.566638 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.566612    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-envoy" pod "6ae4f671-bd04-4b77-9e76-17be23e5f5ee" (UID: "6ae4f671-bd04-4b77-9e76-17be23e5f5ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.566754 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.566636    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26797f1-f34c-44e3-86e9-dd0ec24af003-kube-api-access-v9rvp" pod "a26797f1-f34c-44e3-86e9-dd0ec24af003" (UID: "a26797f1-f34c-44e3-86e9-dd0ec24af003"). InnerVolumeSpecName "kube-api-access-v9rvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.566754 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.566720    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-podinfo" pod "a26797f1-f34c-44e3-86e9-dd0ec24af003" (UID: "a26797f1-f34c-44e3-86e9-dd0ec24af003"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:45.566869 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.566792    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-envoy" pod "a26797f1-f34c-44e3-86e9-dd0ec24af003" (UID: "a26797f1-f34c-44e3-86e9-dd0ec24af003"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.566869 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.566828    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-token" pod "6ae4f671-bd04-4b77-9e76-17be23e5f5ee" (UID: "6ae4f671-bd04-4b77-9e76-17be23e5f5ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.566952 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.566861    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-podinfo" pod "bdcde015-efb5-4689-84f3-6e65620b8a53" (UID: "bdcde015-efb5-4689-84f3-6e65620b8a53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:45.567311 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.567288    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-envoy" pod "bdcde015-efb5-4689-84f3-6e65620b8a53" (UID: "bdcde015-efb5-4689-84f3-6e65620b8a53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.567419 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.567402    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdcde015-efb5-4689-84f3-6e65620b8a53-kube-api-access-kqttt" pod "bdcde015-efb5-4689-84f3-6e65620b8a53" (UID: "bdcde015-efb5-4689-84f3-6e65620b8a53"). InnerVolumeSpecName "kube-api-access-kqttt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.568121 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.568103    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-token" pod "bdcde015-efb5-4689-84f3-6e65620b8a53" (UID: "bdcde015-efb5-4689-84f3-6e65620b8a53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.612521 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.612480    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xcv5v_e3059381-b63e-4d76-bf2d-ade2250ff36a/server/0.log"
Oct 29 22:16:45.664544 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664515    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664544 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664539    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664544 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664549    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664557    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdcde015-efb5-4689-84f3-6e65620b8a53-istiod-ca-cert\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664565    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664572    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26797f1-f34c-44e3-86e9-dd0ec24af003-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664580    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-credential-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664588    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664596    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664603    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-istio-data\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664611    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqttt\" (UniqueName: \"kubernetes.io/projected/bdcde015-efb5-4689-84f3-6e65620b8a53-kube-api-access-kqttt\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664619    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-workload-socket\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664633    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69fdc\" (UniqueName: \"kubernetes.io/projected/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-kube-api-access-69fdc\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664643    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664652    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664660    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-token\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664667    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f671-bd04-4b77-9e76-17be23e5f5ee-istio-envoy\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664675    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdcde015-efb5-4689-84f3-6e65620b8a53-workload-certs\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664683    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26797f1-f34c-44e3-86e9-dd0ec24af003-istio-podinfo\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.664777 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.664690    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9rvp\" (UniqueName: \"kubernetes.io/projected/a26797f1-f34c-44e3-86e9-dd0ec24af003-kube-api-access-v9rvp\") on node \"ip-10-0-128-85.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.801488 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:45.801450    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dq4xb_fdf98d9e-00a0-4795-a5e6-f494f873fd0d/manager/0.log"
Oct 29 22:16:46.035014 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.034991    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-r8ft2_a11c33aa-7a9a-422f-8697-a36ef4bc30ba/proxy/0.log"
Oct 29 22:16:46.287890 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.287814    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6" event={"ID":"a26797f1-f34c-44e3-86e9-dd0ec24af003","Type":"ContainerDied","Data":"e79c96bf1b8e9a2b36a4d6c0ba87df2dcf988489beb1e566c007cd0d963d536d"}
Oct 29 22:16:46.287890 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.287876    4550 scope.go:122] "RemoveContainer" containerID="3d9d4df4d210aa83747257be2ea6467fcee5bd0b8644e657a3e9845aa18a4f1f"
Oct 29 22:16:46.288436 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.287821    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"
Oct 29 22:16:46.289299 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.289281    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"
Oct 29 22:16:46.289379 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.289311    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx" event={"ID":"bdcde015-efb5-4689-84f3-6e65620b8a53","Type":"ContainerDied","Data":"d99eb7e9dd6416999894271522a8f2e7a121cdcb69e1095659140dec855a6ffe"}
Oct 29 22:16:46.290875 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.290857    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"
Oct 29 22:16:46.290993 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.290875    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6" event={"ID":"6ae4f671-bd04-4b77-9e76-17be23e5f5ee","Type":"ContainerDied","Data":"393ec4c4ee47493e7e29120223402dc90433e9cd94de2be584ca9fe0c2bc19f2"}
Oct 29 22:16:46.297477 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.297461    4550 scope.go:122] "RemoveContainer" containerID="73823e5c9354fa4d74bf0c904d8de9ebf5d3932aeb8063e28f5f1fb38703bcfb"
Oct 29 22:16:46.305073 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.305057    4550 scope.go:122] "RemoveContainer" containerID="bc9aa7797542acc8e82b2e5a187d3224b80f8b23c8c3cda73743ff5e9d938841"
Oct 29 22:16:46.351791 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.351768    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"]
Oct 29 22:16:46.354379 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.354356    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-tkfd6"]
Oct 29 22:16:46.388722 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.388698    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"]
Oct 29 22:16:46.393022 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.392999    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-4bjw6"]
Oct 29 22:16:46.426477 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.426454    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"]
Oct 29 22:16:46.430111 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:46.430086    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-cc2vx"]
Oct 29 22:16:47.987036 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:47.987007    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" path="/var/lib/kubelet/pods/6ae4f671-bd04-4b77-9e76-17be23e5f5ee/volumes"
Oct 29 22:16:47.987476 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:47.987462    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a26797f1-f34c-44e3-86e9-dd0ec24af003" path="/var/lib/kubelet/pods/a26797f1-f34c-44e3-86e9-dd0ec24af003/volumes"
Oct 29 22:16:47.987849 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:47.987836    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" path="/var/lib/kubelet/pods/bdcde015-efb5-4689-84f3-6e65620b8a53/volumes"
Oct 29 22:16:50.044374 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:50.044343    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:57.435866 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:57.435829    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-f5cm2_20102b85-6195-42e4-94ea-873d5af3ad35/manager/0.log"
Oct 29 22:16:57.862309 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:57.862274    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-q94rt_cd3a6ac7-2936-45af-9951-a1002cbeab78/manager/0.log"
Oct 29 22:16:58.257726 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:58.257615    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f6b55dfc4-hv5c5_6a47b6ed-0ade-4b53-ab33-43e52c37ed92/manager/0.log"
Oct 29 22:16:58.448831 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:58.448798    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xcv5v_e3059381-b63e-4d76-bf2d-ade2250ff36a/server/0.log"
Oct 29 22:16:58.635134 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:58.635103    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dq4xb_fdf98d9e-00a0-4795-a5e6-f494f873fd0d/manager/0.log"
Oct 29 22:16:58.855683 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:58.855656    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-r8ft2_a11c33aa-7a9a-422f-8697-a36ef4bc30ba/proxy/0.log"
Oct 29 22:16:59.602932 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:16:59.602897    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5569c899b4-gr72l_195d1fda-1571-42d4-9c8e-7ecfebdaeade/maas-api/0.log"
Oct 29 22:17:01.961903 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:01.961877    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-2tx5c_877d3ba8-a37b-4317-83a7-81d41b1a7842/postgres/0.log"
Oct 29 22:17:03.720423 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:03.720372    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-779bff956b-4cdnx_3a9b6803-3648-4ea1-95b4-61a8b1245a7b/manager/0.log"
Oct 29 22:17:04.141618 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:04.141575    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-57f5c66455-wqzgt_09c103da-69b2-47c3-a35e-2dc18f6efee7/manager/0.log"
Oct 29 22:17:05.560905 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:05.560874    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_mcp-gateway-controller-6c98b995c5-kczzc_c6436350-e6cf-4ef9-a0ac-2cc72b34fb0b/mcp-controller/0.log"
Oct 29 22:17:07.149620 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:07.149505    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wrfmq_64621438-8bc0-40ce-8fe5-1129114a5fbf/discovery/0.log"
Oct 29 22:17:07.374571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:07.374541    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zlz8n_34be22aa-35ff-4fb0-90d8-f2a523511916/discovery/0.log"
Oct 29 22:17:07.759613 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:07.759570    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64545848b5-gp7tk_9b0e0326-d34a-4c53-a2d9-e4fadc814dbc/kube-auth-proxy/0.log"
Oct 29 22:17:08.140989 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:08.140958    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-b4jg2_c05fddc0-c933-46fa-9dd4-e7881488c8a5/istio-proxy/0.log"
Oct 29 22:17:11.488859 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:17:11.488825    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:17:14.684848 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:14.684814    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:17:21.942040 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:21.942008    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:17:22.118754 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:22.118718    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log" limit=1024
Oct 29 22:17:27.430784 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:27.430757    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7rvg5_9c53ce77-00c9-4509-9650-115117f93d54/global-pull-secret-syncer/0.log"
Oct 29 22:17:28.386180 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:28.386150    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4hxw6_b49ef0f6-2ac3-4393-b45a-5356110e13a3/konnectivity-agent/0.log"
Oct 29 22:17:28.757753 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:28.757677    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-85.ec2.internal_d33928eb6a1a12294cb01f2c571c879b/haproxy/0.log"
Oct 29 22:17:29.796087 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:29.796053    4550 scope.go:122] "RemoveContainer" containerID="8b4497459c15992a3e0bc3cb6deb10c58c2b682e64971a770d8c0d1a00b50551"
Oct 29 22:17:35.578828 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:35.578797    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-779bff956b-4cdnx_3a9b6803-3648-4ea1-95b4-61a8b1245a7b/manager/0.log"
Oct 29 22:17:35.872097 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:35.872026    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-57f5c66455-wqzgt_09c103da-69b2-47c3-a35e-2dc18f6efee7/manager/0.log"
Oct 29 22:17:38.163971 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:38.163944    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_mcp-gateway-controller-6c98b995c5-kczzc_c6436350-e6cf-4ef9-a0ac-2cc72b34fb0b/mcp-controller/0.log"
Oct 29 22:17:39.009741 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.009708    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"]
Oct 29 22:17:39.010121 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010108    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8f0d50a6-d0cf-44ae-86e2-32558679df0b" containerName="istio-proxy"
Oct 29 22:17:39.010198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010123    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0d50a6-d0cf-44ae-86e2-32558679df0b" containerName="istio-proxy"
Oct 29 22:17:39.010198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010136    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" containerName="istio-proxy"
Oct 29 22:17:39.010198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010153    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" containerName="istio-proxy"
Oct 29 22:17:39.010198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010163    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerName="istio-proxy"
Oct 29 22:17:39.010198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010168    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerName="istio-proxy"
Oct 29 22:17:39.010198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010177    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a26797f1-f34c-44e3-86e9-dd0ec24af003" containerName="istio-proxy"
Oct 29 22:17:39.010198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010182    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26797f1-f34c-44e3-86e9-dd0ec24af003" containerName="istio-proxy"
Oct 29 22:17:39.010198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010188    4550 cpu_manager.go:414] "containerMap: removing container" podUID="73cee1bc-536f-4cc5-8874-7409ae9f739c" containerName="istio-proxy"
Oct 29 22:17:39.010198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010193    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73cee1bc-536f-4cc5-8874-7409ae9f739c" containerName="istio-proxy"
Oct 29 22:17:39.010198 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010201    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2913dd75-7d51-4ee4-b705-f6a7396d75de" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010205    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2913dd75-7d51-4ee4-b705-f6a7396d75de" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010211    4550 cpu_manager.go:414] "containerMap: removing container" podUID="37c48198-27e2-4aa6-86e7-389bd62a2666" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010215    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c48198-27e2-4aa6-86e7-389bd62a2666" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010222    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b8b9de66-29e2-4895-9ebe-b006b55614c2" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010227    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b9de66-29e2-4895-9ebe-b006b55614c2" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010235    4550 cpu_manager.go:414] "containerMap: removing container" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010240    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010246    4550 cpu_manager.go:414] "containerMap: removing container" podUID="08a13f88-6791-4bd9-a15f-55b80bbee84d" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010251    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a13f88-6791-4bd9-a15f-55b80bbee84d" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010257    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010261    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010266    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010270    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010277    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2fbcc15a-7f6f-4742-aabb-916bb609889a" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010281    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fbcc15a-7f6f-4742-aabb-916bb609889a" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010286    4550 cpu_manager.go:414] "containerMap: removing container" podUID="06aa5d20-0931-4e09-98ef-6b7cf4c1a59d" containerName="maas-api"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010290    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06aa5d20-0931-4e09-98ef-6b7cf4c1a59d" containerName="maas-api"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010297    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b27b23e3-07bd-4531-9d64-73d1b66de485" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010300    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b27b23e3-07bd-4531-9d64-73d1b66de485" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010306    4550 cpu_manager.go:414] "containerMap: removing container" podUID="df402fae-12c4-4ed9-9fb1-226a51f66c18" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010310    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df402fae-12c4-4ed9-9fb1-226a51f66c18" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010317    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010321    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010328    4550 cpu_manager.go:414] "containerMap: removing container" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010333    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010343    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5cb930ab-251e-4279-9963-f6d0d02b5b98" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010347    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb930ab-251e-4279-9963-f6d0d02b5b98" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010358    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8cf254b1-d374-4b55-8fec-d04f6b6501c1" containerName="maas-api"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010362    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf254b1-d374-4b55-8fec-d04f6b6501c1" containerName="maas-api"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010368    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7fae4096-9b8a-4938-82b4-098312132fe7" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010372    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fae4096-9b8a-4938-82b4-098312132fe7" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010379    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5a89ba55-981b-475e-a86c-833c547e65d5" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010382    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a89ba55-981b-475e-a86c-833c547e65d5" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010389    4550 cpu_manager.go:414] "containerMap: removing container" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010393    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010400    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b27b23e3-07bd-4531-9d64-73d1b66de485" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010405    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b27b23e3-07bd-4531-9d64-73d1b66de485" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010412    4550 cpu_manager.go:414] "containerMap: removing container" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010416    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010424    4550 cpu_manager.go:414] "containerMap: removing container" podUID="3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d" containerName="maas-api"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010428    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d" containerName="maas-api"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010435    4550 cpu_manager.go:414] "containerMap: removing container" podUID="11f45a3b-d463-4335-b8f5-e43831a38ceb" containerName="payload-pre-processing"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010440    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f45a3b-d463-4335-b8f5-e43831a38ceb" containerName="payload-pre-processing"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010447    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e983d6ab-b49f-4e25-ba45-37f08b410917" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010451    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e983d6ab-b49f-4e25-ba45-37f08b410917" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010459    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a3b59718-adb6-437a-8c5d-3003d93b5203" containerName="maas-api"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010463    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b59718-adb6-437a-8c5d-3003d93b5203" containerName="maas-api"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010468    4550 cpu_manager.go:414] "containerMap: removing container" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010472    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010480    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010484    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010489    4550 cpu_manager.go:414] "containerMap: removing container" podUID="be3e55fd-83b9-4c52-a2a9-28da7d04477e" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010493    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3e55fd-83b9-4c52-a2a9-28da7d04477e" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010499    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8020c6d3-1330-4581-8824-2ca1bdf0bc35" containerName="payload-processing"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010504    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8020c6d3-1330-4581-8824-2ca1bdf0bc35" containerName="payload-processing"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010512    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010516    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" containerName="istio-proxy"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010524    4550 cpu_manager.go:414] "containerMap: removing container" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010528    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main"
Oct 29 22:17:39.010510 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010538    4550 cpu_manager.go:414] "containerMap: removing container" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010542    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010548    4550 cpu_manager.go:414] "containerMap: removing container" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010552    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010558    4550 cpu_manager.go:414] "containerMap: removing container" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010563    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010569    4550 cpu_manager.go:414] "containerMap: removing container" podUID="db332eb5-5ced-43e9-953b-5d94aa37d706" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010573    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db332eb5-5ced-43e9-953b-5d94aa37d706" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010578    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d6233b12-31ad-41d0-bb31-12b2f71161bb" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010582    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6233b12-31ad-41d0-bb31-12b2f71161bb" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010639    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="66cfb49c-2ee3-4afb-b3dd-7e81cb9395cd" containerName="main"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010647    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="09bace6e-b116-4c71-a2c1-1265a83b1608" containerName="main"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010653    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdcde015-efb5-4689-84f3-6e65620b8a53" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010660    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="280967ca-516c-40d1-891f-7c3edb621486" containerName="main"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010667    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="db332eb5-5ced-43e9-953b-5d94aa37d706" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010673    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6233b12-31ad-41d0-bb31-12b2f71161bb" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010679    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0d50a6-d0cf-44ae-86e2-32558679df0b" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010684    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e00a1d-038e-4ed0-8a8c-a85ae64e5452" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010688    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae4f671-bd04-4b77-9e76-17be23e5f5ee" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010692    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26797f1-f34c-44e3-86e9-dd0ec24af003" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010696    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="73cee1bc-536f-4cc5-8874-7409ae9f739c" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010702    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2913dd75-7d51-4ee4-b705-f6a7396d75de" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010706    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c48198-27e2-4aa6-86e7-389bd62a2666" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010712    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b9de66-29e2-4895-9ebe-b006b55614c2" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010717    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd727a21-2709-44be-a77d-f8cb1482e317" containerName="main"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010721    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a13f88-6791-4bd9-a15f-55b80bbee84d" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010726    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f0953b3-a4b1-425f-bf8c-1b0dc847b38c" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010730    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e01274e1-5f68-4edf-bb58-f1ec4dbbf3a9" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010735    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fbcc15a-7f6f-4742-aabb-916bb609889a" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010741    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="06aa5d20-0931-4e09-98ef-6b7cf4c1a59d" containerName="maas-api"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010746    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b27b23e3-07bd-4531-9d64-73d1b66de485" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010751    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="df402fae-12c4-4ed9-9fb1-226a51f66c18" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010757    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1021a25d-cebc-4bad-965d-6afc5bad66f9" containerName="main"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010762    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff86f8c-7f09-43cb-bffc-889eb6e56181" containerName="main"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010768    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb930ab-251e-4279-9963-f6d0d02b5b98" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010774    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf254b1-d374-4b55-8fec-d04f6b6501c1" containerName="maas-api"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010779    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fae4096-9b8a-4938-82b4-098312132fe7" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010784    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a89ba55-981b-475e-a86c-833c547e65d5" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010788    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f97b3ec-8b10-4b46-b8e0-233c3ce8c9c1" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010794    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b27b23e3-07bd-4531-9d64-73d1b66de485" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010799    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="660c3442-f3d1-4eb4-a013-232f82f8d81b" containerName="main"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010803    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c47f2b9-9ea7-44aa-92ad-1a4e38511a5d" containerName="maas-api"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010808    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f45a3b-d463-4335-b8f5-e43831a38ceb" containerName="payload-pre-processing"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010813    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e983d6ab-b49f-4e25-ba45-37f08b410917" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010818    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b59718-adb6-437a-8c5d-3003d93b5203" containerName="maas-api"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010822    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="20bda0fc-d2ac-4435-9e5d-bc6b557e59b0" containerName="main"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010827    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5058fbcb-91cf-44bd-9d4c-34f2acc83f23" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010831    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3e55fd-83b9-4c52-a2a9-28da7d04477e" containerName="istio-proxy"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010836    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8020c6d3-1330-4581-8824-2ca1bdf0bc35" containerName="payload-processing"
Oct 29 22:17:39.012484 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.010842    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc8e407-edc2-4c71-ba8b-f43f53dc8f57" containerName="istio-proxy"
Oct 29 22:17:39.015757 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.015738    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.032457 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.032434    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"]
Oct 29 22:17:39.033833 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.033806    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bzkjc\"/\"kube-root-ca.crt\""
Oct 29 22:17:39.033965 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.033811    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bzkjc\"/\"openshift-service-ca.crt\""
Oct 29 22:17:39.131434 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.131406    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bzkjc\"/\"default-dockercfg-78mjz\""
Oct 29 22:17:39.140562 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.140540    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b5379ab-419c-4bfd-b011-99fe09e35426-sys\") pod \"perf-node-gather-daemonset-jtdtj\" (UID: \"0b5379ab-419c-4bfd-b011-99fe09e35426\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.140639 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.140583    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0b5379ab-419c-4bfd-b011-99fe09e35426-proc\") pod \"perf-node-gather-daemonset-jtdtj\" (UID: \"0b5379ab-419c-4bfd-b011-99fe09e35426\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.140692 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.140676    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b5379ab-419c-4bfd-b011-99fe09e35426-lib-modules\") pod \"perf-node-gather-daemonset-jtdtj\" (UID: \"0b5379ab-419c-4bfd-b011-99fe09e35426\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.140736 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.140720    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0b5379ab-419c-4bfd-b011-99fe09e35426-podres\") pod \"perf-node-gather-daemonset-jtdtj\" (UID: \"0b5379ab-419c-4bfd-b011-99fe09e35426\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.140772 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.140753    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jclcr\" (UniqueName: \"kubernetes.io/projected/0b5379ab-419c-4bfd-b011-99fe09e35426-kube-api-access-jclcr\") pod \"perf-node-gather-daemonset-jtdtj\" (UID: \"0b5379ab-419c-4bfd-b011-99fe09e35426\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.241872 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.241839    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0b5379ab-419c-4bfd-b011-99fe09e35426-proc\") pod \"perf-node-gather-daemonset-jtdtj\" (UID: \"0b5379ab-419c-4bfd-b011-99fe09e35426\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.242334 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.241881    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b5379ab-419c-4bfd-b011-99fe09e35426-lib-modules\") pod \"perf-node-gather-daemonset-jtdtj\" (UID: \"0b5379ab-419c-4bfd-b011-99fe09e35426\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.242334 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.241939    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0b5379ab-419c-4bfd-b011-99fe09e35426-podres\") pod \"perf-node-gather-daemonset-jtdtj\" (UID: \"0b5379ab-419c-4bfd-b011-99fe09e35426\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.242334 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.241982    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jclcr\" (UniqueName: \"kubernetes.io/projected/0b5379ab-419c-4bfd-b011-99fe09e35426-kube-api-access-jclcr\") pod \"perf-node-gather-daemonset-jtdtj\" (UID: \"0b5379ab-419c-4bfd-b011-99fe09e35426\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.242334 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.241987    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b5379ab-419c-4bfd-b011-99fe09e35426-lib-modules\") pod \"perf-node-gather-daemonset-jtdtj\" (UID: \"0b5379ab-419c-4bfd-b011-99fe09e35426\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.242334 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.242080    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0b5379ab-419c-4bfd-b011-99fe09e35426-proc\") pod \"perf-node-gather-daemonset-jtdtj\" (UID: \"0b5379ab-419c-4bfd-b011-99fe09e35426\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.242334 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.242088    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b5379ab-419c-4bfd-b011-99fe09e35426-sys\") pod \"perf-node-gather-daemonset-jtdtj\" (UID: \"0b5379ab-419c-4bfd-b011-99fe09e35426\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.242334 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.242114    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0b5379ab-419c-4bfd-b011-99fe09e35426-podres\") pod \"perf-node-gather-daemonset-jtdtj\" (UID: \"0b5379ab-419c-4bfd-b011-99fe09e35426\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.242334 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.242134    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b5379ab-419c-4bfd-b011-99fe09e35426-sys\") pod \"perf-node-gather-daemonset-jtdtj\" (UID: \"0b5379ab-419c-4bfd-b011-99fe09e35426\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.264923 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.264863    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jclcr\" (UniqueName: \"kubernetes.io/projected/0b5379ab-419c-4bfd-b011-99fe09e35426-kube-api-access-jclcr\") pod \"perf-node-gather-daemonset-jtdtj\" (UID: \"0b5379ab-419c-4bfd-b011-99fe09e35426\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.327799 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.327775    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:39.624571 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:39.624546    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"]
Oct 29 22:17:39.627171 ip-10-0-128-85 kubenswrapper[4550]: W1029 22:17:39.627118    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0b5379ab_419c_4bfd_b011_99fe09e35426.slice/crio-7064b5c601df214e176c365ef22e56e54acf5e288bcbf7f8b00d4d1581154569 WatchSource:0}: Error finding container 7064b5c601df214e176c365ef22e56e54acf5e288bcbf7f8b00d4d1581154569: Status 404 returned error can't find the container with id 7064b5c601df214e176c365ef22e56e54acf5e288bcbf7f8b00d4d1581154569
Oct 29 22:17:40.429881 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:40.429844    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-f5cm2_20102b85-6195-42e4-94ea-873d5af3ad35/manager/0.log"
Oct 29 22:17:40.544043 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:40.544012    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj" event={"ID":"0b5379ab-419c-4bfd-b011-99fe09e35426","Type":"ContainerStarted","Data":"0b3fb79302739f68a945d17a844c6599e1dcb6a6b0d9974a5eea30c6eb20349c"}
Oct 29 22:17:40.544229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:40.544064    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj" event={"ID":"0b5379ab-419c-4bfd-b011-99fe09e35426","Type":"ContainerStarted","Data":"7064b5c601df214e176c365ef22e56e54acf5e288bcbf7f8b00d4d1581154569"}
Oct 29 22:17:40.544229 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:40.544093    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:40.565573 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:40.565544    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-f5cm2_20102b85-6195-42e4-94ea-873d5af3ad35/manager/0.log"
Oct 29 22:17:40.584513 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:40.584460    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj" podStartSLOduration=2.5844443999999998 podStartE2EDuration="2.5844444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:40.583428256 +0000 UTC m=+1876.740553754" watchObservedRunningTime="2026-10-29 22:17:40.5844444 +0000 UTC m=+1876.741569899"
Oct 29 22:17:40.938425 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:40.938390    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-q94rt_cd3a6ac7-2936-45af-9951-a1002cbeab78/manager/0.log"
Oct 29 22:17:41.159236 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:41.159203    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-q94rt_cd3a6ac7-2936-45af-9951-a1002cbeab78/manager/0.log"
Oct 29 22:17:41.172978 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:41.172954    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f6b55dfc4-hv5c5_6a47b6ed-0ade-4b53-ab33-43e52c37ed92/manager/0.log"
Oct 29 22:17:41.215970 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:41.215902    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xcv5v_e3059381-b63e-4d76-bf2d-ade2250ff36a/server/0.log"
Oct 29 22:17:41.358916 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:41.358887    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f6b55dfc4-hv5c5_6a47b6ed-0ade-4b53-ab33-43e52c37ed92/manager/0.log"
Oct 29 22:17:41.383157 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:41.383120    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sl2vf_6d50eb73-ad36-4be8-90f1-c39479159f97/node-exporter/0.log"
Oct 29 22:17:41.398033 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:41.398010    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xcv5v_e3059381-b63e-4d76-bf2d-ade2250ff36a/server/0.log"
Oct 29 22:17:41.496674 ip-10-0-128-85 kubenswrapper[4550]: E1029 22:17:41.496587    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8bd7" containerName="extractor"
Oct 29 22:17:41.503470 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:41.503448    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sl2vf_6d50eb73-ad36-4be8-90f1-c39479159f97/kube-rbac-proxy/0.log"
Oct 29 22:17:41.628448 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:41.628421    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sl2vf_6d50eb73-ad36-4be8-90f1-c39479159f97/init-textfile/0.log"
Oct 29 22:17:41.758820 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:41.758745    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dq4xb_fdf98d9e-00a0-4795-a5e6-f494f873fd0d/manager/0.log"
Oct 29 22:17:41.801898 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:41.801870    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-r8ft2_a11c33aa-7a9a-422f-8697-a36ef4bc30ba/proxy/0.log"
Oct 29 22:17:41.918640 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:41.918611    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dq4xb_fdf98d9e-00a0-4795-a5e6-f494f873fd0d/manager/0.log"
Oct 29 22:17:41.956048 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:41.956023    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-r8ft2_a11c33aa-7a9a-422f-8697-a36ef4bc30ba/proxy/0.log"
Oct 29 22:17:42.869010 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:42.868984    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f07465f-53a1-4caa-8f87-7e1bd060baca/prometheus/0.log"
Oct 29 22:17:42.982911 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:42.982888    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f07465f-53a1-4caa-8f87-7e1bd060baca/config-reloader/0.log"
Oct 29 22:17:43.097259 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:43.097233    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f07465f-53a1-4caa-8f87-7e1bd060baca/thanos-sidecar/0.log"
Oct 29 22:17:43.213065 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:43.213002    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f07465f-53a1-4caa-8f87-7e1bd060baca/kube-rbac-proxy-web/0.log"
Oct 29 22:17:43.331244 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:43.331207    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f07465f-53a1-4caa-8f87-7e1bd060baca/kube-rbac-proxy/0.log"
Oct 29 22:17:43.353097 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:43.353073    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-f5cm2_20102b85-6195-42e4-94ea-873d5af3ad35/manager/0.log"
Oct 29 22:17:43.443437 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:43.443411    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f07465f-53a1-4caa-8f87-7e1bd060baca/kube-rbac-proxy-thanos/0.log"
Oct 29 22:17:43.529266 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:43.529204    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-f5cm2_20102b85-6195-42e4-94ea-873d5af3ad35/manager/0.log"
Oct 29 22:17:43.557250 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:43.557225    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f07465f-53a1-4caa-8f87-7e1bd060baca/init-config-reloader/0.log"
Oct 29 22:17:43.681984 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:43.681953    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-x77sz_658d3ab9-05e5-416e-9d80-0ab77a49885b/prometheus-operator/0.log"
Oct 29 22:17:43.796475 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:43.796402    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-x77sz_658d3ab9-05e5-416e-9d80-0ab77a49885b/kube-rbac-proxy/0.log"
Oct 29 22:17:43.849754 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:43.849717    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-q94rt_cd3a6ac7-2936-45af-9951-a1002cbeab78/manager/0.log"
Oct 29 22:17:43.916658 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:43.916624    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-dh2g2_9174ad9f-ec68-430f-990d-75c2c26817e1/prometheus-operator-admission-webhook/0.log"
Oct 29 22:17:44.089557 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:44.089519    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-q94rt_cd3a6ac7-2936-45af-9951-a1002cbeab78/manager/0.log"
Oct 29 22:17:44.099595 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:44.099568    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f6b55dfc4-hv5c5_6a47b6ed-0ade-4b53-ab33-43e52c37ed92/manager/0.log"
Oct 29 22:17:44.151760 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:44.151735    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xcv5v_e3059381-b63e-4d76-bf2d-ade2250ff36a/server/0.log"
Oct 29 22:17:44.324956 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:44.324922    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f6b55dfc4-hv5c5_6a47b6ed-0ade-4b53-ab33-43e52c37ed92/manager/0.log"
Oct 29 22:17:44.367957 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:44.367891    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xcv5v_e3059381-b63e-4d76-bf2d-ade2250ff36a/server/0.log"
Oct 29 22:17:44.703648 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:44.703562    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dq4xb_fdf98d9e-00a0-4795-a5e6-f494f873fd0d/manager/0.log"
Oct 29 22:17:44.753819 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:44.753788    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-r8ft2_a11c33aa-7a9a-422f-8697-a36ef4bc30ba/proxy/0.log"
Oct 29 22:17:44.875095 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:44.875062    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dq4xb_fdf98d9e-00a0-4795-a5e6-f494f873fd0d/manager/0.log"
Oct 29 22:17:44.926925 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:44.926899    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-r8ft2_a11c33aa-7a9a-422f-8697-a36ef4bc30ba/proxy/0.log"
Oct 29 22:17:46.335833 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:46.335796    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-f5cm2_20102b85-6195-42e4-94ea-873d5af3ad35/manager/0.log"
Oct 29 22:17:46.490359 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:46.490327    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-f5cm2_20102b85-6195-42e4-94ea-873d5af3ad35/manager/0.log"
Oct 29 22:17:46.567846 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:46.567797    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-jtdtj"
Oct 29 22:17:46.847706 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:46.847671    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-q94rt_cd3a6ac7-2936-45af-9951-a1002cbeab78/manager/0.log"
Oct 29 22:17:47.081463 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:47.081422    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-q94rt_cd3a6ac7-2936-45af-9951-a1002cbeab78/manager/0.log"
Oct 29 22:17:47.112463 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:47.112365    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f6b55dfc4-hv5c5_6a47b6ed-0ade-4b53-ab33-43e52c37ed92/manager/0.log"
Oct 29 22:17:47.164830 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:47.164802    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xcv5v_e3059381-b63e-4d76-bf2d-ade2250ff36a/server/0.log"
Oct 29 22:17:47.312266 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:47.312231    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f6b55dfc4-hv5c5_6a47b6ed-0ade-4b53-ab33-43e52c37ed92/manager/0.log"
Oct 29 22:17:47.374919 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:47.374838    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xcv5v_e3059381-b63e-4d76-bf2d-ade2250ff36a/server/0.log"
Oct 29 22:17:47.744842 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:47.744749    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dq4xb_fdf98d9e-00a0-4795-a5e6-f494f873fd0d/manager/0.log"
Oct 29 22:17:47.804030 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:47.803997    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-r8ft2_a11c33aa-7a9a-422f-8697-a36ef4bc30ba/proxy/0.log"
Oct 29 22:17:47.924424 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:47.924396    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dq4xb_fdf98d9e-00a0-4795-a5e6-f494f873fd0d/manager/0.log"
Oct 29 22:17:47.977409 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:47.977385    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-r8ft2_a11c33aa-7a9a-422f-8697-a36ef4bc30ba/proxy/0.log"
Oct 29 22:17:49.778415 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:49.778384    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-cqnpv_526e5ad0-b682-4946-9a79-c83b34eed94b/download-server/0.log"
Oct 29 22:17:54.625710 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:54.625670    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x5mwg_84b63148-f00f-4506-b07d-9c0c0777034c/dns/0.log"
Oct 29 22:17:54.739688 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:54.739661    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x5mwg_84b63148-f00f-4506-b07d-9c0c0777034c/kube-rbac-proxy/0.log"
Oct 29 22:17:54.856309 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:54.856272    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6bf42_09d024f0-dc43-4c99-81dd-ec9a2c126e86/dns-node-resolver/0.log"
Oct 29 22:17:56.508352 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:56.508288    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5689cfdf84-bdbdz_ea13b764-00ed-4482-943c-337fb11ae9e5/registry/0.log"
Oct 29 22:17:56.890866 ip-10-0-128-85 kubenswrapper[4550]: I1029 22:17:56.890838    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rx95w_a3cbc0d0-b4ec-4e1d-85c2-99f4642cd2f2/node-ca/0.log"